How to scrape Amazon product data with Python

Choose Amazon search or product data first

To scrape Amazon use the endpoint matching your input. A search term belongs in the search endpoint; an ASIN belongs in the product endpoint. They return different shapes, so one parsing loop won't work for both.

ScraperAPI's Amazon search documentation describes GET https://api.scraperapi.com/structured/amazon/search. Supply query and your api_key. Its JSON example has a results array of product objects, plus next_pages and explore_more_items arrays.

The product endpoint documentation describes GET https://api.scraperapi.com/structured/amazon/product with asin and api_key. It returns a product object, with fields such as name, product_information, brand, images and feature_bullets in the documented sample.

The product sample doesn't establish a guaranteed top-level price, rating or availability field. Don't invent one because it would make a convenient CSV column. Search results document price, price_string and price_symbol, but those can still be absent on individual rows.

Set up a small, bounded request

You'll need Python and the requests package. Set your own ScraperAPI key in an environment variable called SCRAPERAPI_KEY; don't paste it into a committed script, notebook output or CSV. Our ScraperAPI review explains the service and its wider use cases.

python -m pip install requests

The script below makes one search request and writes a local CSV. It doesn't loop over Amazon pages or fetch every returned ASIN. Those would be additional requests with additional credit costs.

ScraperAPI documents internal retries lasting up to 70 seconds. We use a 10-second connection timeout and 90-second read timeout so the normal read window exceeds that internal retry period. The tuple is our client configuration, not a vendor guarantee that every request finishes within 90 seconds. A read timeout also isn't a total runtime cap.

One search request, then a CSV export

This example follows the documented search response. It was checked offline with illustrative fixtures, not against a paid live API response. Treat the output as a collection snapshot, not a price guarantee.

import csv
import os
from datetime import datetime, timezone
from pathlib import Path

import requests

SEARCH_ENDPOINT = "https://api.scraperapi.com/structured/amazon/search"
COLUMNS = ["retrieved_at", "market", "query", "page", "asin",
           "name", "url", "price", "price_symbol"]

def spreadsheet_text(value):
    text = "" if value is None else str(value)
    if text.lstrip().startswith(("=", "+", "-", "@")):
        return "'" + text
    return text

def search_amazon(query, market="com", page=1):
    key = os.environ.get("SCRAPERAPI_KEY")
    if not key:
        raise RuntimeError("Set SCRAPERAPI_KEY before running.")
    params = {"api_key": key, "query": query, "tld": market,
              "country_code": "us", "page": page,
              "output_format": "json"}
    try:
        response = requests.get(SEARCH_ENDPOINT, params=params,
                                timeout=(10, 90))
    except requests.RequestException:
        # Request exceptions may include the URL and its secret query key.
        raise RuntimeError("Request failed; check usage before retrying.") from None
    status = response.status_code
    if status == 404:
        raise RuntimeError("Target not found (404); this may consume credits.")
    if status != 200:
        raise RuntimeError(f"Request stopped with HTTP {status}; no automatic retry.")
    try:
        data = response.json()
    except ValueError:
        raise RuntimeError("Expected JSON; inspect a sanitized response.") from None
    if not isinstance(data, dict) or not isinstance(data.get("results"), list):
        raise RuntimeError("Unexpected search schema; results must be a list.")
    return data

def rows_from_search(data, query, market="com", page=1):
    if not isinstance(data, dict) or not isinstance(data.get("results"), list):
        raise ValueError("Expected a search object with results[].")
    stamp = datetime.now(timezone.utc).isoformat()
    seen = set()
    for item in data["results"]:
        if not isinstance(item, dict) or item.get("type") != "search_product":
            continue
        asin = item.get("asin")
        if not isinstance(asin, str) or not asin.strip():
            continue
        identity = (market, asin)
        if identity in seen:
            continue
        seen.add(identity)
        price = item.get("price")
        # Preserve a genuine zero; missing or unexpected price stays blank.
        if isinstance(price, bool) or not isinstance(price, (int, float)):
            price = ""
        row = {"retrieved_at": stamp, "market": market, "query": query,
               "page": page, "asin": asin, "name": item.get("name"),
               "url": item.get("url"), "price": price,
               "price_symbol": item.get("price_symbol")}
        yield {k: (v if k == "price" else spreadsheet_text(v))
               for k, v in row.items()}

def write_csv(rows, filename):
    with Path(filename).open("w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=COLUMNS)
        writer.writeheader()
        writer.writerows(rows)

if __name__ == "__main__":
    term = "ceramic mug"
    payload = search_amazon(term)
    write_csv(rows_from_search(payload, term), "amazon-search.csv")

The script fails closed when it doesn't recognize the response shape. It also avoids printing request exceptions, because the documented endpoint puts the API key in query parameters. An exception containing the prepared URL can leak that key even when your source code uses an environment variable.

Our spreadsheet-text helper prefixes potentially executable text values. Quoting a CSV cell alone doesn't stop spreadsheet formula interpretation. Numeric prices stay numeric; absent prices stay blank. Some spreadsheet programs handle imports differently, so review the import settings before opening data you don't trust.

What the offline checks cover

We checked parsing with a documentation-shaped fixture containing duplicate ASINs, missing fields, a non-product entry and a text value beginning with a formula character. Separate checks cover an empty result list, unexpected schema and simulated HTTP or request failures. These are code checks, not evidence of scraping success on Amazon.

Deduplication here is limited to one exported batch and the pair of market plus ASIN. If you combine pages or collection dates later, decide whether you want one row per product or one row per observation. Deduplicating the whole dataset by ASIN would erase the price history you might be trying to collect.

Markets, country targeting and pagination

tld selects the Amazon marketplace, such as com or co.uk. country_code describes geographic targeting. They aren't interchangeable. The product documentation says ASINs are market-specific, so don't assume an identifier and fields collected in one market transfer unchanged to another.

The example is deliberately US-oriented: marketplace com and country code us. If you change the marketplace, review the targeting and your plan's supported locations rather than changing one string and assuming the rest remains correct.

The search endpoint documents a page parameter and returns next_pages in its sample. For a larger export, add an explicit maximum page count and a request budget before introducing a loop. Stop on empty results and log the page number. Don't blindly follow every URL forever.

A successful response can still contain a product without a price. It may be unavailable, presented differently or absent from the parsed fields; the response alone doesn't establish which explanation is correct. Keep the cell blank and retain the original JSON when you need to investigate.

Fetch a known product without assuming the search schema

For an ASIN lookup, call the product endpoint with asin, tld and the appropriate targeting, using the same timeout and sanitized-error approach. Read the returned object rather than looking for results[].

Start with fields actually shown in the product documentation: name, product_information.asin, brand, and images. Nested lists and dictionaries need their own handling; don't let Python stringify a large object into a CSV cell without deciding whether that format suits your downstream system.

The page's opening mentions public reviews, but its sample has an empty reviews list. We wouldn't promise a complete review archive based on that. If your task requires reviews, offers or a guaranteed field, verify the dedicated documentation and sample for that task before committing to a large collection.

Credits, errors and the cost of retries

The official credit-cost documentation lists Amazon at a base cost of five credits per request. Parameters and protection handling can change the actual cost. The response header sa-credit-cost reports the charge for a request; don't add rendering or premium options merely because another tutorial does.

The current pricing page lists Hobby at $49 per month, with 100,000 credits and 20 concurrent threads. Its $44.10 monthly equivalent is billed annually. The page separately describes a seven-day trial with 5,000 credits and no card, and a free-plan FAQ with different allowances. Those aren't one combined free balance.

The status-code documentation distinguishes these cases:

ScraperAPI says successful 200 and 404 responses are charged, while failed 500 requests aren't. Its credit documentation also warns that cancelling your side before it has had 70 seconds can incur a charge. Therefore a client timeout must not trigger an unlimited immediate retry loop. Check usage first when completion is uncertain.

When a different approach makes sense

If you're learning HTML parsing, a local saved page and BeautifulSoup can teach the extraction step without a stream of live requests. A browser automation project may suit a permitted task requiring interaction, but it adds another system to maintain. Neither approach automatically solves access restrictions or grants permission to collect data.

Read the site's terms, assess privacy and intellectual-property obligations, and collect only the public data appropriate to your task. A scraping provider's technical capability isn't a legal opinion. This guide doesn't cover account-only content or bypassing access controls.

For Google search data rather than Amazon products, use our Google scraping guide. If search results are the whole workload, our SERP API comparison addresses that narrower selection.

Make one useful export before scaling

Collect a small search response, inspect the JSON and open the CSV safely. Only then add pagination, product lookups or a schedule with an explicit credit budget. The extra requests are easy to write; making their output trustworthy takes more care.

Start with ScraperAPI's trial