How to scrape Google search results in 2026

What “scrape Google search results” actually means

People use the phrase for three different jobs:

1. Pull the HTML (or JSON) of a Google results page for a keyword, location and device.
2. Parse titles, URLs, snippets and SERP features into a spreadsheet or database.
3. Track ranks over time, which is really job 1 + 2 on a schedule.

Only the first two belong in a scraping guide. Rank tracking has ready-made tools; building it from scrapes only pays off when you need custom logic. We cover that trade-off in rank tracking API: build or buy.

Rules, ToS and what this guide is not

Google's Terms of Service restrict automated access to Search. Local privacy law, your employer’s policy and the site’s robots rules can also apply. This article is not legal advice. We describe technical options people use; you decide whether they fit your use case and counsel.

Also: scraping doesn’t grant you rights to republish Google’s content wholesale. Keep datasets for analysis you are allowed to run.

Why DIY Google scrapers fail

A weekend script often looks fine once:

# illustrative only, not a recipe we ran against Google
import requests
from bs4 import BeautifulSoup

html = requests.get(
    "https://www.google.com/search",
    params={"q": "best rank tracker", "hl": "en", "gl": "us"},
    headers={"User-Agent": "Mozilla/5.0"},
    timeout=30,
).text
soup = BeautifulSoup(html, "html.parser")

Then Google returns a consent interstitial, a CAPTCHA, or markup with empty result nodes. Datacenter IPs get throttled. Layout classes change. You bolt on residential proxies, a headless browser and retry logic, and suddenly you maintain infrastructure, not a parser.

That’s the gap a scraping API fills: you keep the parse step; someone else keeps the browser farm warm.

Option A, ScraperAPI (our pick for most how-tos)

ScraperAPI sits between your code and the target URL. You request a page. It rotates proxies, retries blocks, and can render JavaScript. For Google and a few other large sites it can also return structured fields instead of raw HTML (see their current docs for endpoint names, they change naming more often than the credit rules).

Prices (from our November 2026 review, re-check live)

Figures below are what ToolVerdict recorded on ScraperAPI’s public pricing page for the ScraperAPI review. Re-check scraperapi.com/pricing before you buy:

Annual billing took 10% off Hobby–Business on the page we checked. Unused credits did not roll over. Normal pages cost 1 credit; Amazon 5; Google or Bing 25; LinkedIn 30.

So ScraperAPI is convenient for mixed scraping (sites + some Google). It is not the cheapest pure SERP feed, see cheapest SERP API when Google is 100% of the workload.

Illustrative request (fixtures + docs shape, no live run)

We did not call ScraperAPI for this article (no credits spent). The snippet follows the documented API pattern: GET https://api.scraperapi.com/ with api_key and url query params (ScraperAPI docs). Extra flags such as country_code or render belong in the same query string when you need them. Confirm names on the current docs page before shipping.

Our published ScraperAPI review notes you are charged for successful responses (including some non-200 cases such as 404 on their rules). Failed auth or hard errors should not silently burn a Google-sized credit pack. Still check the dashboard after your first trial calls.

# ILLUSTRATIVE, syntax-checked only. Not run against ScraperAPI.
# Set SCRAPERAPI_KEY in the environment for a real call.
# Or set SCRAPERAPI_FIXTURE=/path/to/saved.html to parse offline.
import os
import sys
from pathlib import Path

import requests

API_KEY = os.environ.get("SCRAPERAPI_KEY")
FIXTURE = os.environ.get("SCRAPERAPI_FIXTURE")
google_results = "https://www.google.com/search?q=best+rank+tracker&hl=en&gl=us"

if FIXTURE:
    html = Path(FIXTURE).read_text(encoding="utf-8", errors="replace")
else:
    if not API_KEY:
        sys.exit("Set SCRAPERAPI_KEY or SCRAPERAPI_FIXTURE")
    try:
        resp = requests.get(
            "https://api.scraperapi.com/",
            params={
                "api_key": API_KEY,
                "url": google_results,
                # optional geo hint, confirm in current ScraperAPI docs:
                "country_code": "us",
            },
            timeout=90,
        )
    except requests.Timeout:
        sys.exit("ScraperAPI request timed out")
    except requests.RequestException as exc:
        sys.exit(f"ScraperAPI request failed: {exc}")

    if resp.status_code == 401:
        sys.exit("Unauthorized, check SCRAPERAPI_KEY")
    if resp.status_code == 429:
        sys.exit("Rate limited / thread limit, back off and retry")
    if resp.status_code >= 400:
        sys.exit(f"HTTP {resp.status_code}: {resp.text[:200]}")

    html = resp.text

# Parse yourself. Google markup changes; do not hard-code one CSS path.
print("html_bytes", len(html))

Validated here: Python 3 syntax (py_compile) and the request shape against public docs. Not validated here: live HTTP, credit deduction, or today's Google HTML.

Open the ScraperAPI trial and burn the first credits on your real queries

Pros and cons for Google scraping

Pros - One integration for Google and ordinary sites - Proxies, retries and optional JS rendering included - Trial large enough to measure credits per request on your keywords

Cons - 25 credits per Google search makes small plans pricey vs dedicated SERP APIs - Credits don’t roll over on the plans we logged - You still own parsing and storage

Option B, Dedicated SERP APIs

If every request is a Google SERP, compare providers that bill per thousand searches (Serper, SerpApi, DataForSEO and others). Our SERP API price table puts those on a common $/1,000 footing. None of them is a ToolVerdict affiliate top pick on this how-to page; ScraperAPI remains the CTA because it matches the “scrape Google + other sites” job.

Option C, Official Google APIs

Google’s Custom Search JSON API is the sanctioned programmable search product, with its own quotas and billing. It is not a full anonymous SERP dump, and setup differs from scraping. Use it when you need a compliant search API for a product feature, not when you are trying to mirror the public results page pixel-for-pixel.

Option D, Don’t scrape: use a rank tracker

Tracking 500 to 5,000 keywords daily is almost always cheaper in a tracker than in DIY scrapes once you count engineering time. SE Ranking Core is $129/mo ($103.20 billed annually) for 2,000 daily keywords plus research and audits (subscription page, figures from our November 2026 pricing pass). Fourteen-day trial, no card.

Compare rank trackers → · Try SE Ranking →

A sane workflow

1. Write down the fields you need (position, URL, title, ads, local pack…).
2. Run 20–50 real queries through a trial API and note credits or $ per SERP.
3. Parse into a schema; store the raw payload for a week so layout changes don’t erase history.
4. Add backoff, idempotency keys and alerts before you schedule hourly jobs.
5. Revisit whether a tracker or SERP API is cheaper after a month of real volume.

How we put this guide together

ScraperAPI plan and credit figures come from the vendor pricing capture used in our ScraperAPI review (November 2026). Re-open ScraperAPI pricing before you buy in case credits or plan names moved. We did not run ScraperAPI or scrape Google for this page; the sample is docs-shaped and syntax-checked, with an offline fixture path. Method: how we test.

Collecting retail products instead? Our Amazon Python scraping tutorial covers its different JSON schemas and a bounded CSV export.

The bottom line

Scraping Google in 2026 is rarely about a clever CSS selector. It’s about who eats the proxy bill. ScraperAPI is our pick here when you want one API for Google plus other sites and 25 credits per search is acceptable. Pure SERP shops win on unit price. Rank trackers win when you never needed the HTML.

Start the ScraperAPI trial