Requests vs Headless Browser for Web Scraping: We Fetched 30 Pages Three Ways

Most posts on this topic are written by companies that sell browsers. Unsurprisingly, they conclude you need a browser.

We sell a fetch API, so you should read this one with the same suspicion. To make that easier, we did the thing almost nobody in this debate does: we ran the same 30 pages through all three methods and wrote down what came back.

Here's the short version:

• Plain `requests` is enough far more often than vendors admit. It got 10 out of 10 static pages, fastest, for free.
• A headless browser is a rendering tool, not an unblocking tool. Default headless Chrome did worse than `requests` on bot-protected sites: 1 out of 10.
• The question that actually matters is what state the page needs to be in. And, if an LLM is reading the result, how many tokens you're willing to pay for it.

Need clean page content for an agent? Try TinyFetch. It's free for up to 1,000 URLs a day (as of 27 Oct 2026).

How We Tested Requests, Headless Chrome, and TinyFetch

We picked 30 public pages, 10 of each kind:

• Static: Wikipedia, the Python docs, MDN, GOV.UK, Hacker News, Paul Graham's essays and similar pages that send their content in the first HTML response.
• JS-heavy: apps that build the page in the browser, such as an Ashby job board, Swagger UI, Solana Explorer, Uniswap, Excalidraw and Airbnb search.
• Bot-protected: Zillow, Ticketmaster, Walmart, G2, Indeed, Glassdoor, Best Buy, Crunchbase, Etsy and Nike, all public pages behind well-known bot defences.

Each page was fetched three ways:

1. `requests` + BeautifulSoup: a GET with a normal Chrome user agent, then `get_text()`.
2. Headless Chrome: Playwright driving the installed Chrome, headless, waiting for the network to go quiet, then reading `innerText`.
3. TinyFetch: our Fetch API, live (no cache), returning markdown.

The rule that matters most: a fetch only counted as a success if the page's real content came back. Not a 200 status, the content. For every page we picked a phrase that only appears once the real content loads ("Circulating Supply" on Solana Explorer, "sqft" on Zillow) and checked for it.

That distinction turned out to be most of the story.

Web Scraping Results from 30 Test Pages

Run on 27 October 2026 from one laptop on home broadband, one request per page per method. TinyFetch timings are server-side; the other two include the laptop's network. Bot defences change with IP, time of day and volume, so treat the bot-protected row as one afternoon's snapshot, not a rate.

Now, what it means.

1. Plain requests deserves more respect

On static pages, `requests` was perfect, the fastest, and costs nothing. Wikipedia came back in well under a second.

This is the boring truth most "you need a headless browser" posts skip: a huge share of the web still sends its content in the first HTML response. Docs, government sites, blogs, news, forums. If you can see the data in View Source, a browser is pure overhead.

Apify, to its credit, is one of the few vendors that says this out loud. In one of their own tests, the same query took 10.4 seconds over plain HTTP and timed out after 379 seconds in browser mode.

Our take: start every new source with View Source. If the data is there, use `requests` and move on with your life.

2. On JS-heavy pages, requests gets nothing (and tells you it got something)

`requests` went 0 for 10 on JS-heavy pages. That part isn't surprising.

What's dangerous is how it failed. Every one of those requests returned 200 OK. The body was an empty app shell: "You need to enable JavaScript to run this app", or a page frame with no data in it. Uniswap sent 88 KB of HTML containing 14 tokens of actual text.

If your pipeline checks the status code and moves on, it will happily store thousands of empty pages and report success. This is the one point we agree with the "always render" camp on: a half-rendered 200 is a failure that looks like a success. Where we disagree is the fix. Rendering everything isn't the answer. Checking the content is.

Headless Chrome got all 10, at roughly four to six times the time of a plain request. That's the honest case for a browser: the content genuinely doesn't exist until JavaScript builds it.

TinyFetch got 5 of 10 here, and we want to be straight about it:

• Airbnb came back as placeholder cards ("Fake title 1…"), the skeleton the page shows before data arrives.
• Swagger UI rendered with every section collapsed, so the detail we checked for wasn't on screen.
• Workable returned empty content, and Solana Explorer rate-limited us.

The pattern is pages that need waiting or clicking before the content exists. That's exactly where you should reach for a browser you control, or an agent that can click. TinyFetch is built for reading what a page shows on load. It isn't trying to be a browser, and you shouldn't pretend it is.

3. Headless Browsers and Bot Detection: What Our Test Found

This is the result we'd most like browser vendors to sit with.

On bot-protected pages, default headless Chrome succeeded once in ten tries. Plain `requests` got four.

• Zillow served full HTML to `requests` and a "Press & Hold" challenge to the browser, after an 18-second wait.
• Walmart gave `requests` 1.56 MB of real HTML. The browser got "Robot or human?"
• Ticketmaster returned a 403 to the browser.

It makes sense once you think about it. A default headless browser announces itself: it literally identifies as `HeadlessChrome`, and bot systems look at far more than the user agent. A simple HTTP request with a normal header can look less suspicious than an automated browser. Browser Use's stealth benchmark found something similar: headless Chromium on a datacenter IP got past bot protection only 2% of the time.

So when you see claims like "HTTP scrapers are ineffective for 68% of sites" or "95%+ success against anti-bot systems", ask for the test set. We couldn't find one behind either number.

Our take on bot walls is less exciting than a vendor's: nothing we tested gets through reliably, us included. TinyFetch also got 4 of 10, a different four (it was the only method to get both Indeed and Ticketmaster, and it failed G2, Crunchbase, Etsy and Glassdoor). When a site works hard to keep automated traffic out, the durable answer is usually its official API, a licensed data feed, or permission, not an escalating arms race of stealth plugins. We've written about why stealth plugins fail if you want the longer version.

4. LLM Token Usage: Raw HTML vs. Extracted Text

If a person reads your scraped pages, page size barely matters. If an LLM reads them, it's most of your bill.

• Walmart: 521,000 tokens of HTML held 1,700 tokens of text.
• Nike: headless Chrome's rendered page came to 639,000 tokens. TinyFetch returned the product grid in about 725.
• Zillow: 258,000 tokens of HTML. TinyFetch returned the listings as about 512 tokens of markdown, in 0.7 seconds.

Even "clean" output differs. On static pages, BeautifulSoup's `get_text()` gave us a median of 1,747 tokens because it keeps every nav link and footer line. TinyFetch's markdown came in at 1,309, because it strips the boilerplate. (It occasionally strips a line you wanted: on two pages our check phrase was a heading it treated as chrome, which is why we report its static score as 9 of 10, not 10.)

Our take: if an agent is the reader, "which method gets the page" is only half the question. The other half is "how much of what comes back is the page." Dumping raw HTML into a context window is the most expensive way to scrape there is. We measured this in more depth in 80% of your web fetch returns junk.

How to Choose the Right Web Scraping Method

1. View source. Is the data in the HTML? Use `requests`. You're done. Don't let anyone talk you into a browser.
2. Is there a JSON endpoint in the Network tab? Call it directly with `requests`. It's faster and cleaner than any rendering.
3. Does the content appear on load, without clicks? Use a fetch API that renders, or a headless browser. If an LLM is the reader, prefer the one that returns clean text.
4. Does it need clicks, scrolling, a login or a form? Use a browser you drive, like Playwright or TinyBrowser, or an agent that can do the clicking.
5. Is it behind a serious bot wall? Stop and look for an official API, a licensed feed, or ask. A default headless browser will usually make it worse.

And at every step: check that the content is there, not just that you got a 200.

What this looks like in code

Here's the escalation pattern we'd actually ship. Try the cheapest thing first, check for real content, and only escalate when you have to.

import requests
from bs4 import BeautifulSoup
from tinyfish import TinyFish

client = TinyFish()  # reads TINYFISH_API_KEY

def has_content(text: str, must_contain: str) -> bool:
    return must_contain.lower() in text.lower()

def fetch_page(url: str, must_contain: str) -> dict:
    # 1. Plain HTTP: free and fast when the HTML has the data
    try:
        r = requests.get(url, timeout=20, headers={"User-Agent": "Mozilla/5.0"})
        text = BeautifulSoup(r.text, "lxml").get_text(" ", strip=True)
        if r.ok and has_content(text, must_contain):
            return {"method": "requests", "text": text}
    except requests.RequestException:
        pass

    # 2. TinyFetch: renders when needed, returns clean markdown
    result = client.fetch.get_contents(urls=[url])
    if result.results and has_content(result.results[0].text, must_contain):
        return {"method": "fetch", "text": result.results[0].text}

    # 3. Needs clicks or waiting: hand off to a browser or an agent
    return {"method": "escalate", "text": None}

The `must_contain` check is the important line. It's what turns "we got a 200" into "we got the page".

When to Use Requests, Headless Browsers, or a Fetch API

• Use `requests` for static pages and JSON endpoints. It's still the right default for most of the web.
• Use a headless browser when content only exists after JavaScript, waiting or interaction, and when you need to control the page yourself.
• Use a fetch API like TinyFetch when an agent or LLM is reading the result and you'd rather not own rendering, retries and clean-up. It matched `requests` on static pages and returned a fraction of the tokens.
• Use none of the above on sites that are actively keeping automated traffic out.

The industry argument is framed as "requests vs headless browser", as if you pick a team. You don't. You pick per page, you verify the content, and you pay for rendering only when the page needs it.

Building an agent that reads the web? Start free with TinyFish: TinySearch and TinyFetch are free to start, and TinyBrowser and TinyAgent run on the same API key from your Wallet, which starts with $8.

Keep reading

• How We Evaluate Fetch Quality for AI Agents
• 80% of Your Web Fetch Returns Junk
• Production-Grade Web Fetching for AI Agents
• Headless Browser vs AI Agents
• Which TinyFish API Should You Use?

Follow TinyFish for new guides and builds: X (@Tiny_Fish) · LinkedIn · Discord