API Features
Basic Request
To scrape a web page using the scrapestack API, simply use the API's base endpoint and append the URL you would like to scrape, as well as your API access key, as GET parameters. There is also a series of optional parameters you can choose from. Below you will find an example request used to scrape the URL https://apple.com.
Example API Request:
Request Parameters:
| Object | Description |
|---|---|
access_key | [Required] Specify your unique API access key to authenticate with the API. Your API access key can be found in your account dashboard. |
url | [Required] Specify the URL of the web page you would like to scrape. |
render_js | [optional] Set to 0 (off, default) or 1 (on) depending on whether or not to render JavaScript on the target web page. JavaScript rendering is done using a Google Chrome headless browser. |
keep_headers | [optional] Set 0 (off, default) or 1 (on) depending on whether or not to send currently active HTTP headers to the target URL with your API request and have the API return these headers along with your API response. |
proxy_location | [optional] Specify the 2-letter code of the country you would like to us as a proxy geolocation for your scraping API request. Supported countries differ by proxy type, please refer to the Proxy Locations section for details. |
premium_proxy | [optional] Set 0 (off, default) or 1 (on) depending on whether or not to enable premium residential proxies for your scraping request. Please note that a single premium proxy API request is counted as 25 API requests. |
Example API Response:
If your scraping request was successful, the API will respond with the raw HTML data of your target web page URL. If you have enabled HTTP headers, your API response will also contain the HTTP headers sent along with your original API request.
API response shortened: Please note that the API response above has been shortened for readability purposes. To see the entire API resonse, please click the "Run API Request" button in the Example API Request section or sign up for an access key.
JavaScript Rendering
Some web pages render essential page elements using JavaScript, which means that some content is not present (and therefore not scrapable) with the initial page load. With the render_js parameter enabled, the scrapestack API is capable of accessing the target web using a headless browser (Google Chrome) and allows JavaScript page elements to render before delivering the final scraping result.
To enable JavaScript rendering, simply append the render_js HTTP GET parameter to your API request URL and set it to 1. By default, this parameter is set to 0 (off).
Example API Request:
Example API Response:
To see an API response, please click the "Run API Request" button above or sign up for an API access key.
HTTP Headers
The scrapestack API will accept HTTP headers and pass them through to the target web page and the final API response if the keep_headers HTTP GET parameter is set to 1. By default, this parameter is set to 0 (off).
Below you will find an example API request (Bash using the "curl" command) that contains an HTTP X-Header. If this request is executed, the specified header will be sent to the target web page and returned in the final API response.
Example Bash (curl) Request with HTTP header:
Unsupported HTTP Headers: Although most HTTP headers are supported by the API, there are some that cannot be processed. Please find a list of unsupported HTTP headers below:
content-encodingcontent-length
Proxy Locations
The scrapestack API is making use of a pool of 35+ million IP addresses worldwide. By default, the API will auto-rotate IP addresses in a way that the same IP address is never used twice in a row.
Across both standard and premium proxies, the scrapestack API supports more than 100 global geolocations your scraping request can be sent from. Using the API's proxy_location HTTP GET parameter you can choose a specific country by its 2-letter country code for your scraping request. Please find an example API request below, which specifies au (for Australia) as proxy location.
Example API Request:
Example API Response:
To see an API response, please click the "Run API Request" button above or sign up for an API access key.
Standard Proxies - Supported Countries:
For standard (datacenter) proxies, the API currently supports a total of 77 global geolocations. You can download a full list of supported countries and 2-letter country codes using the following link: locations-standard-proxy.csv
Premium Proxies - Supported Countries:
For premium (residential) proxies, the API currently supports a total of 38 global geolocations. You can download a full list of supported countries and 2-letter country codes using the following link: locations-premium-proxy.csv
Premium Proxies
By default, the scrapestack will always use standard (datacenter) proxies for scraping requests. While these proxies are the most common proxies used on the internet, they are also much more likely to get blocked on an IP-basis when attempting to scrape data.
For customers subscribed to the Professional Plan or higher, the scrapestack API allows access to premium (residential) proxies, which are associated with real residential addresses and therefore much less likely to get blocked while scraping data on the web. To perform your request using premium proxies, simply append the API's premium_proxy HTTP GET parameter and set it to 1.
Example API Request:
Example API Response:
To see an API response, please click the "Run API Request" button above or sign up for an API access key.
Premium Proxy Requests: Please note that a single request using premium proxies will be charged as 25 API requests, while single requests using standard proxies are charged as only 1 API request.
HTTP POST/PUT Requests
The scrapestack API also offers a way of scraping forms or API endpoints directly by supporting API requests via HTTP POST/PUT. Below you will find relevant examples that can be used both via POST and PUT.
Example Request - HTTP POST:
Example Request - HTTP POST with Form Data: