You already know how to write a scraper. The question isn't whether you can — it's whether you should.
For a lot of web data problems, Scrapy or Playwright is still the right answer. Cheaper, faster, predictable. But there's a class of problems where selectors fail structurally, not just occasionally — and patching them is a treadmill that never stops. This article is a framework for knowing which problem you're actually facing before you commit to either path.
Kadoa's 2026 enterprise scraping report documented one team tracking 14 marketplaces that hit 9 site structure changes in a single quarter. That's real, and it's a genuine argument for moving away from selectors. But it's not the whole story.
For pages that are static, publicly accessible, and structurally stable — a Wikipedia article, a government data portal, a public product catalog that hasn't changed its HTML in two years — a well-written Scrapy spider is fast, free beyond compute costs, and completely deterministic. An AI agent on those same pages is slower and costs money per step. The economics don't favor it.
The honest version of this comparison starts with where selectors work, not where they fail.
The problem isn't that selectors are fragile. It's that they're fragile in specific structural ways that compound as sites get more sophisticated.
Limit 1: JavaScript rendering. A traditional HTTP request gets you the raw HTML. On a React or Vue SPA, that raw HTML contains almost nothing useful — the actual content arrives via client-side JS execution. Playwright solves this, but now you're managing a headless browser process, which is a different infrastructure class than a simple HTTP scraper.
Limit 2: Anti-bot systems. Modern sites with strict automation requirements deploy TLS fingerprinting, behavioral analysis, and detection challenges that identify headless browser patterns. This is now the norm on any commercially significant site. Bypassing these requires infrastructure-level browser identity management and proxy routing — not script patches.
Limit 3: Login walls and session state. A selector can't log in, handle 2FA, maintain session cookies across tab navigations, or re-authenticate when a session expires mid-run. Any target that requires authentication is structurally outside the reach of traditional scraping.
Limit 4: Multi-step workflows. Price comparison, filter application, form submission, paginated search with dynamic state — these require reasoning about what to do next, not pattern matching on a known HTML structure. A selector script for a 10-step workflow is a state machine you wrote by hand. It breaks at step 7 when the site changes step 3.
The right tool is a function of the site, not the data you want from it. Here's how to classify any target before you build:
| Tier | Site Characteristics | What Breaks Scrapy | Recommended Tool |
|---|---|---|---|
| Tier 1 | Static HTML, public, stable structure | Nothing | Scrapy / BeautifulSoup |
| Tier 2 | JS-rendered, public, no login | JS rendering | Playwright + Fetch API |
| Tier 3 | Sites with strict requirements OR requiring authentication | Infrastructure reliability / session | Browser API (browser_profile: stealth) |
| Tier 4 | Multi-step workflow, dynamic state, authenticated | Reasoning required | Web Agent |
Tier 1 example: A government statistical database with stable HTML tables. Scrapy. Write it once, run it indefinitely.
Tier 2 example: A public e-commerce catalog with React rendering but no authentication or strict automation requirements. Playwright works. Alternatively, TinyFish's Fetch API handles JS-rendered pages without managing your own headless browser infrastructure. Same result, less overhead.
Tier 3 example: E-commerce platforms and business portals with strict automation requirements — sites that need infrastructure-level handling or authenticated access on accounts you control.
Tier 4 example: "Find all open purchase orders in our supplier portal (accounts your team manages), check the expected delivery date, and flag any that are more than 5 days late." This requires login, navigation, cross-page state tracking, and conditional logic. A selector script for this is a maintenance project, not a one-time build.
The argument for traditional scrapers usually focuses on setup cost: Scrapy is free, you know how to use it, done in a day. The argument against focuses on maintenance cost — which is real but often underestimated.
Industry pricing guides put in-house scraper maintenance at 5–10 hours per week of engineering time once you're running multiple sources (source: Tendem.ai 2026 web scraping cost guide). At a conservative $100/hour fully-loaded engineering cost, 10 hours/week is $52,000/year in maintenance labor — before accounting for silent failures where bad data goes unnoticed for days.
A 2025 McGill University study (Cohen & Hage-Youssef, "Generative AI for Data Scraping," SSRN) tested AI extraction across 3,000 pages on Amazon, Cars.com, and Upwork. AI-powered methods achieved up to 98.4–100% accuracy in testing even when page structures changed, with costs as low as $0.0004 per page. Setup time drops from weeks to hours compared to traditional selector-based approaches.
The cost comparison at 50 sources (annualized):
| Approach | Annual cost (50 sources) | Calculation |
|---|---|---|
| In-house Scrapy + maintenance | ~$26,000–$52,000 | $100/hr eng × 5–10 hrs/week × 52 weeks |
| TinyFish Web Agent | ~$27,400 | 50 sources × 100 URLs × 365 days × $0.015/step |
| TinyFish Fetch API (Tier 2) | $0 | Free on all plans — 0 credits, TinyFish owns the stack |
The Web Agent cost is in the same range as in-house maintenance — the break-even argument isn't price, it's reliability and engineering time freed. Search and Fetch are free on all plans — rate-limited by plan tier (Free: 5 searches/min, 25 fetches/min). Failed fetches are never charged. ongoing maintenance overhead. The right tool choice depends on which tier your sources actually fall into.
The crossover point: if you're maintaining more than a handful of selectors and your sources update layout more than once a year, the total cost of ownership — including debugging time and silent failures — is comparable to or lower than in-house maintenance. Below that threshold, Scrapy wins on pure economics.
The concrete failure scenario: a button's class name changes from btn-submit to button-primary. Your Playwright scraper silently returns a 200 with no useful data. Two days later someone notices the data hasn't updated. An engineer spends half a day tracing the failure, fixes the selector, redeploys. At 50 sources with four layout changes per year per source: 200 debug sessions annually, budgeted or not.
The right approach isn't "rip and replace everything with web agents." It's a tiered escalation that matches infrastructure complexity to actual site complexity.
Step 1 — Fetch API for Tier 2 sites. If you're currently running Playwright to handle JS rendering, TinyFish's Fetch API is a simpler drop-in: POST a URL, get back clean markdown, JSON, or HTML. No browser process to manage, no infrastructure to maintain. Failed URLs aren't charged.
import os, requests
# Fetch API: HTTP POST, returns markdown/json/html — no browser process required
response = requests.post(
"https://api.fetch.tinyfish.ai",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"], "Content-Type": "application/json"},
json={
"urls": ["https://example.com/product/123"],
"format": "markdown",
"links": True
}
)
print(response.json())Step 2 — Browser API for Tier 3 sites. If you need full browser control — anti-bot handling, session state, your own Playwright scripts against a managed remote browser — the Browser API gives you a CDP WebSocket connection with sub-250ms cold start. Zero changes to existing Playwright code.
import os, requests
from playwright.sync_api import sync_playwright
# Step 1: create remote session
resp = requests.post(
"https://api.browser.tinyfish.ai",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"], "Content-Type": "application/json"},
json={}
)
cdp_url = resp.json()["cdp_url"]
# Step 2: connect with your existing Playwright code — zero changes
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp(cdp_url)
page = browser.new_page()
page.goto("https://your-target-site.com")
# .. your existing scraping logic unchangedStep 3 — Web Agent for Tier 4 sites. When the task requires navigation decisions, multi-step workflows, or authenticated access, the Web Agent handles the full sequence. Natural language goal in, structured JSON out.
import os, requests
# Web Agent: goal-driven, handles login/navigation/extraction end-to-end
response = requests.post(
"https://agent.tinyfish.ai/v1/automation/run-sse",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"], "Content-Type": "application/json"},
json={
"url": "https://your-supplier-portal.com",
"goal": "Log in and return all open POs with expected delivery date and current status"
},
stream=True,
)
for line in response.iter_lines():
if line:
print(line.decode("utf-8"))All three products share the same API key and credit pool — start with Fetch, escalate to Browser or Agent as your targets get more complex, no migration required.
The comparison is honest if it includes the cases where web agents are the wrong choice:
Low volume, stable public pages. If you're scraping 100 static URLs twice a month, a Scrapy script costs essentially nothing. A web agent at $0.015/step for those same 100 pages is $1.50/run. Neither is expensive, but the scraper is simpler and faster.
Budget-constrained with no anti-bot requirement. At small scale on cooperative sites, the cost difference between free open-source tools and a credit-based agent API is real. The right answer is Scrapy until you hit a structural wall.
Real-time streaming data at very high frequency. If you need thousands of requests per minute from cooperative APIs or static pages, a web agent adds latency and cost that isn't justified. High-frequency ticker data, RSS feeds, public REST APIs — these are not web agent use cases.
You just need HTML parsing. If the page renders and you need to parse its contents without any interaction, Fetch API or even simple requests + BeautifulSoup handles it more cheaply than a full agent step.
The rule: use the simplest tool that can handle the site's structural complexity. Escalate when that tool's structural limits become maintenance overhead.
Yes. Playwright is the standard tool for connecting to the TinyFish Browser API — same code, different infrastructure. The skill transfers directly.
For e-commerce and SaaS portals, 2–4 layout changes per year per source is a reasonable baseline. Static government and academic sites change far less. Your maintenance cost scales with site churn, not just site count.
Scrapy on a cloud VM: ~$0.0001–0.001 per page (compute only). TinyFish Fetch and Search APIs are free on all plans. TinyFish Web Agent: $0.015 per step for multi-step tasks. For simple JS-rendered pages the Fetch API costs nothing; for multi-step authenticated workflows, Scrapy often can't complete the task at all.
For JS-rendered pages: yes, and it's free. POST a URL, get back markdown, JSON, or HTML — no browser process, no selector maintenance. The trade-off is Scrapy's crawling framework; for full-site crawls you'd manage the URL queue yourself, or use the Web Agent for multi-page traversal.
TinyFish's infrastructure layer handles reliability at the execution level. If a run encounters issues, the response includes detailed status information — SITE_BLOCKED, TASK_FAILED, or TIMEOUT — so you can handle each case explicitly in your code rather than getting silent failures. The streaming_url in the response gives you a live browser replay for debugging.
No credit card. No setup. Run your first operation in under a minute.
