Google Search Results scraping in 2026 is less reliable and more expensive because Google no longer honors the unofficial num=100 parameter, while automated Google queries remain against Google’s policies. For dependable SEO decisions, prioritize Search Console and approved data providers over direct SERP scraping.
The SEO teams that get dependable answers are not the ones collecting the most SERP HTML. They are the ones that know which data must come from Search Console, which needs local rank tracking, and which is not worth paying to collect.
num=100 change.Most of what ranks for this query was written against a Google that no longer exists. The code still runs, which is worse than it failing outright, because you get a page back and parse zero results from it and spend an afternoon debugging your selectors instead of your request. In this article, we’ll explore what changed, how to build a request Google treats as ordinary, and how to pull organic results, People Also Ask, and rankings out of a page that rewrites its own markup.
The &num=100 parameter returned a hundred results in one request. Rank trackers ran on it. Google switched it off in mid-September 2025 with no announcement, and when pressed, a spokesperson told Search Engine Land the company had never formally supported it. That was true and unhelpful in equal measure, given it had worked for over a decade. The effect showed up instantly in Search Console. An analysis of 319 properties found 87.7 percent lost impressions and 77.6 percent lost unique ranking terms, because the parameter had been inflating both with results no human scrolled to.
For you this is arithmetic. Ten results per request means ten times the requests for the same depth, which means ten times the IPs, the bandwidth, and the block exposure. Any cost model carried over from the old parameter is off by an order of magnitude, and any tutorial that still shows num=100 was written before this and has not been checked since.
The second change gets less attention and matters more. Guides that tell you to skip scraping and use Google’s own API are pointing at a closed door. Google’s documentation states the Custom Search JSON API is not available to new customers, and existing ones have until January 1, 2027 to migrate. It was never a real substitute anyway. It returns results from a Programmable Search Engine you configure rather than the live index, gives you 100 free queries per day, charges $5 per thousand after that, and caps hard at 10,000 per day.
So the official route is shut, and the efficient route is gone. Fetching the search endpoint is what is left.
Before Google parses a single header, it has already seen your TLS handshake. Python’s default HTTP stack produces a JA3 fingerprint that no browser produces, and no amount of User-Agent spoofing hides it. This is why tutorials that hand you a realistic header dictionary and a Chrome/62 User-Agent still get CAPTCHAs, and why “add better headers” is advice that stopped working years ago.
The fix is a client that replicates a browser’s handshake. curl_cffi wraps curl-impersonate and reproduces Chrome’s TLS signature and HTTP/2 frame ordering:
from curl_cffi import requests
PROXY = “http://USERNAME:PASSWORD@PROXY_HOST:PORT”
def fetch_serp(query, start=0, gl=”us”, hl=”en”):
response = requests.get(
“https://www.google.com/search”,
params={“q”: query, “start”: start, “gl”: gl, “hl”: hl},
impersonate=”chrome”,
proxies={“http”: PROXY, “https”: PROXY},
timeout=20,
)
response.raise_for_status()
return response.text
The second signal is the exit IP, and it carries more weight than everything else combined. A datacenter address running repeated search queries collects a CAPTCHA within a handful of requests regardless of how good the fingerprint is.Residential proxies route through real consumer connections, which is the difference between traffic that looks unusual and traffic that looks like a person.
The IP also decides your geography, and this trips up more rank tracking scripts than any other single thing. The gl parameter changes the flavor of the results Google returns. It does not change the location Google infers, which comes from where the request originates. Asking for gl=de over a US connection gives you a US SERP with German seasoning, not the German SERP. If you need Berlin’s results, you need a Berlin IP. Proxyon covers 195 countries with city-level selection on residential and bills pay-as-you-go rather than by subscription, which matters when your request volume just multiplied by ten.
Three habits keep this alive at volume. Rotate the exit IP per request rather than per session, since Google fingerprints by address far more aggressively than by header. Randomize the delay between requests instead of using a fixed interval, because perfect regularity is itself a tell. And treat a 429 or a redirect to sorry/index as a soft failure: back off exponentially and retry on a different IP rather than hammering the same one until it is burned.
Google rotates its CSS class names. Any guide handing you .tF2Cxc, .yuRUbf, or .related-question-pair is giving you code with a shelf life measured in weeks, and this is the real reason scrapers “randomly stop working.”
Anchoring to Google’s hidden accessibility headings is the usual proposed fix, and it is only half right. A selector that matches on the literal text “Search Results” works in English and returns nothing the moment you set hl to anything else, which means it silently breaks exactly when you start collecting international data.
The durable pattern is an h3 inside a result anchor. That relationship has held through every layout change of the last few years, it does not depend on class names, and it does not depend on interface language:
from parsel import Selector
def parse_organic(html):
sel = Selector(html)
results, seen = [], set()
for anchor in sel.xpath(“//a[.//h3]”):
url = anchor.xpath(“./@href”).get()
title = anchor.xpath(“.//h3//text()”).get()
if not url or not url.startswith(“http”) or not title:
continue
if url in seen:
continue
seen.add(url)
results.append({“title”: title, “url”: url})
return results
The dedup matters because sitelinks nest additional anchors inside a parent result, and without it your rank numbers drift after the first result that has them.
Validate the output rather than trusting it. A page that parses to zero results almost always means you were served a consent interstitial or a CAPTCHA, not that the query had no matches. Check the count on every response, push anything empty into a retry queue on a fresh IP, and log the raw HTML on failure. That log costs nothing and saves hours the next time Google shifts its markup.
These are the least stable parts of the page and the most commonly broken code in published tutorials. Rather than pinning a class, extract candidates by shape and filter them. Questions end in a question mark and sit in the main results column, which is enough to identify them without depending on any attribute Google controls:
def parse_questions(html):
sel = Selector(html)
seen = set()
for node in sel.xpath(“//div[@role=’heading’]//text()”):
text = node.get().strip()
if text.endswith(“?”) and len(text) > 10 and text not in seen:
seen.add(text)
return list(seen)
This returns more than you asked for on some pages and less on others, which is honest about what these sections are. Treat the output as candidates and validate them downstream instead of assuming a fixed count.
Rank is the start offset plus the index within the response, not the index alone. Getting this wrong is common now that every page holds ten results instead of a hundred:
def find_rank(query, domain, max_pages=3):
position = 0
for page in range(max_pages):
for result in parse_organic(fetch_serp(query, start=page * 10)):
position += 1
if domain in result[“url”]:
return position, result
return None, None
Cap max_pages deliberately. With the old parameter, depth was nearly free. It is not anymore, and for most teams anything past the first two or three pages is data you are paying real money to collect and will never act on.
Scraping Google in 2026 comes down to three things, and clever parsing is the least of them. Your handshake has to look like a browser, your exit IP has to sit in the country whose results you are asking for, and your selectors have to survive both a redesign and a language change. Decide your collection depth before you write anything, because that number now drives your entire cost in a way it never did when a hundred results came back in one request. And when the scraper breaks, read the response body before you touch the parser, because nine times out of ten you were handed a CAPTCHA rather than a page.
You can technically retrieve and parse Google Search Results, but Google states that machine-generated traffic, including scraping results for rank-checking and other automated Google Search access without express permission, violates its spam policies and Terms of Service. For most businesses, the safer operational choice is to use Google Search Console for first-party website performance and a vetted third-party rank-tracking or SERP-data provider for defined reporting needs. Review the provider’s data methodology, geographic coverage, reliability, support model, and compliance posture before making it part of your reporting stack.
The num=100 parameter no longer reliably returns 100 Google Search Results in one request. Google stopped honoring the unofficial parameter broadly in mid-September 2025, which forced many rank-tracking workflows to paginate results in smaller sets. That change increased the cost and complexity of collecting rankings beyond the first page. If you need a top-100 snapshot, confirm how your chosen rank-tracking or SERP-data provider handles pagination, result depth, errors, and incomplete responses instead of assuming older scraping logic still works.
The best alternative to scraping Google Search Results depends on the data you need. Google Search Console is best for impressions, clicks, click-through rate, average position, queries, pages, countries, and devices for a site you own. A local rank tracker is best for city, ZIP code, map-pack, and location-specific visibility. A third-party SERP-data provider is best when you need structured results, SERP-feature monitoring, or a programmatic reporting feed. Use manual browser searches for qualitative review of the live customer experience.
Google Search Result parsers often return zero results because the response is not a normal search-results page, not because the query has no matches. The returned HTML may be a consent screen, CAPTCHA, unusual-traffic page, experiment variant, localized layout, sign-in state, or error response. Before changing selectors, log and inspect the response body, page title, response size, locale, and visible text. Build a validation rule that treats unexpected zero-result responses as data-quality exceptions rather than valid ranking data.
You should track rankings beyond the top 10 Google results only when deeper positions support a specific SEO decision. Positions 11 to 30 are often useful for finding near-miss pages that deserve a targeted content refresh, stronger internal links, improved category copy, or better search-intent alignment. Tracking positions 31 to 100 is useful for migrations, competitive research, major category opportunities, and diagnostic work, but it rarely deserves weekly monitoring for every keyword. Use a tiered keyword list so cost and reporting effort match business value.
