Your requests.get() returns HTML. You parse it with BeautifulSoup. The data isn't there.
The page loaded fine in Chrome. But requests got the shell before JavaScript ran — the content you need loaded afterward. Or your script worked fine for weeks, then started failing silently on some URLs in your list but not others.
This is the state of Python web scraping in 2026. The fundamentals haven't changed. What's changed is which sites require something beyond the basics — and what the modern options look like when they do.
This guide covers the full stack: when requests is still the right answer, when you need a real browser, and when managed infrastructure removes the problems that otherwise become full-time engineering work.
As with all data collection: respect each site's terms of service and robots.txt. The examples here cover legitimate use cases — price monitoring, research, internal pipelines, publicly available data.
The Python scraping ecosystem itself is stable. requests, BeautifulSoup, lxml, Playwright, Scrapy — these tools work the same as they did two years ago.
What's changed is the web:
More JavaScript rendering. A larger portion of commercial sites now load their primary content via JavaScript after the initial response. The HTML that requests receives is often just a framework shell — a few <div> containers and a script tag, no actual content.
More dynamic sessions. Sites that previously served data to any request now require cookies, tokens, or interaction sequences before they return useful content.
More infrastructure requirements at scale. Running Python scrapers at volume — hundreds of domains, thousands of URLs — introduces infrastructure complexity that wasn't present at small scale: managing browser servers, handling proxy rotation for IP diversity, debugging failures that appear at scale but not locally.
None of this makes requests obsolete. Most scraping tasks are still Tier 1 or Tier 2 — static HTML or moderately JS-rendered pages at moderate volume. For those, Python's existing tools are exactly right.
The change is that the threshold where you need something beyond a Python script arrives sooner, and when it does, the options are clearer than they used to be.
Every site you want to scrape falls into one of three tiers. Choosing the right tool requires knowing which tier you're in.
| Tier | What the site is | Right Python approach | When to move up |
|---|---|---|---|
| 1 | Static HTML, or has an API | `requests` + `BeautifulSoup` | When content loads via JS |
| 2 | JavaScript-rendered, no strict automation requirements | Playwright (local) or TinyFish Fetch API | When infra becomes the bottleneck at scale |
| 3 | Strict automation requirements, or managed infra needed | TinyFish Fetch or Agent API | When workflow needs multi-step decisions |
Most sites are Tier 1. Start there. Only move up when the simpler tool has a specific reason not to work.
If the site is static HTML, stop here. requests is the right tool.
Check for a JSON API first. Many sites that look like they require scraping actually load data from API endpoints your script can call directly. Browser devtools → Network → filter by fetch or application/json. Direct API calls are faster, cleaner, and more stable than HTML parsing.
For sites where the HTML contains the data:
import requests
from bs4 import BeautifulSoup
from typing import Generator
def scrape_static_site(url: str) -> list[dict]:
headers = {"User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0)"}
r = requests.get(url, headers=headers, timeout=15)
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")
results = []
for item in soup.select(".listing-item"):
title_el = item.select_one(".title")
price_el = item.select_one(".price")
if title_el and price_el:
results.append({
"title": title_el.text.strip(),
"price": price_el.text.strip(),
"url": item.get("href", ""),
})
return resultsFor high-volume static crawls (10K+ URLs), Scrapy adds scheduling, retry logic, deduplication, and output pipeline management that are tedious to build manually. But for smaller volumes, requests in a loop is sufficient.
Signal that you need Tier 2: You run the script, parse the HTML, and the containers are there but the data isn't. Or you see <div id="root"></div> or <div id="app"></div> with nothing inside — that's a JavaScript framework waiting to render.
Content that loads after JavaScript runs requires a real browser. The requests library fetches what the server sends before any JavaScript executes. If your target content is populated by client-side code — React, Vue, Angular, dynamic APIs called from the page — it won't be in that initial response.
Two options, given equal treatment:
Option A: Playwright — local control, full flexibility
import asyncio
from playwright.async_api import async_playwright
from typing import Optional
async def scrape_js_page(url: str, wait_selector: str) -> Optional[str]:
async with async_playwright() as p:
browser = None
try:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto(url, wait_until="networkidle", timeout=30000)
await page.wait_for_selector(wait_selector, timeout=10000)
content = await page.inner_text(wait_selector)
return content
except Exception as e:
print(f"Error fetching {url}: {e}")
return None
finally:
if browser:
await browser.close()
result = asyncio.run(scrape_js_page("https://example.com", ".product-grid"))Playwright is the right choice when you need direct browser control: custom headers, specific interaction sequences, local debugging, CI pipeline tests. You manage the browser process.
Option B: TinyFish Browser API — managed infrastructure, same Playwright code
Connect to TinyFish's managed browser instead of launching one locally. Your selectors and logic stay identical:
import asyncio
import os
from tinyfish import TinyFish
from playwright.async_api import async_playwright
from typing import Optional
async def scrape_js_managed(url: str, selector: str = "body") -> Optional[str]:
"""Extract JS-rendered content using TinyFish managed browser."""
async with async_playwright() as p:
browser = None
try:
session = TinyFish().browser.sessions.create()
browser = await p.chromium.connect_over_cdp(session.cdp_url)
page = await browser.new_page()
await page.goto(url, wait_until="networkidle", timeout=30000)
content = await page.inner_text(selector)
return content
except Exception as e:
print(f"Error loading {url}: {e}")
return None
finally:
if browser:
await browser.close()
content = asyncio.run(scrape_js_managed("https://example.com", ".product-grid"))The automation logic is unchanged. TinyFish manages browser servers, proxy routing, and reliability — browser cold starts under 250ms (source: tinyfish.ai).
For local development, Playwright. For scalable extraction without browser server management, TinyFish Fetch.
For more on when Playwright is the right local choice and where it runs into limits at scale, see Scraping Dynamic Websites: When Playwright Breaks.
The gap between a working local script and a reliable production pipeline isn't a code quality problem — it's an infrastructure gap.
At scale, your Python code is correct. What fails is everything around it:
TinyFish removes this entire layer. Your Python code stays clean — one API call, same interface whether you're running 10 requests or 10,000. The infrastructure complexity shifts from your codebase to TinyFish's.
For straightforward extraction with managed infrastructure:
import requests
import os
def managed_fetch(url: str) -> dict:
"""Fetch with TinyFish managed infrastructure."""
response = requests.post(
"https://api.fetch.tinyfish.ai",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"]},
json={"urls": [url], "format": "markdown"},
timeout=60
)
if not response.ok:
return {"url": url, "content": None, "ok": False}
results = response.json().get("results", [])
return {"url": url, "content": results[0].get("text") if results else None, "ok": True}For goal-directed workflows — multi-step navigation, authenticated access, conditional logic:
import requests
import os
from typing import Optional
def run_agent(url: str, goal: str) -> Optional[dict]:
"""Run a goal-directed extraction agent."""
response = requests.post(
"https://agent.tinyfish.ai/v1/automation/run",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"]},
json={"url": url, "goal": goal},
timeout=120
)
if not response.ok:
return None
data = response.json()
# COMPLETED means the run finished, not necessarily that the goal succeeded
if data.get("status") != "COMPLETED" or data.get("result") is None:
return None
return data["result"]
result = run_agent(
url="https://supplier.example.com",
goal="Find all product listings on the current page. Extract name, SKU, price, and stock status. Return as JSON array."
)TinyFish achieves up to 85% success rate on sites with strict automation requirements and a browser cold start under 250ms. For production pipelines, the infrastructure overhead shifts from your codebase to TinyFish's.
For running Python extraction at scale across hundreds of URLs simultaneously, see How to Crawl a List of URLs at Scale.
1. Does the site have a public API or RSS feed?
→ YES: Call the API with requests. Don't scrape what you can query.
2. Does the HTML from requests contain the data you need?
→ YES: requests + BeautifulSoup. Done.
→ NO (JS-rendered): Continue to step 3.
3. Do you need managed infrastructure (no browser servers to run, no proxies to configure)?
→ NO + local dev: Playwright
→ YES: TinyFish Fetch API → scales without infrastructure overhead
→ NO (production/scale): TinyFish Fetch API
4. Does the workflow require authentication, multi-step navigation,
or conditional decisions between steps?
→ YES: TinyFish Web Agent
→ NO: TinyFish Fetch APIInstall dependencies and set your API key:
pip install requests
export TINYFISH_API_KEY=sk-tinyfish-your-key-hereFetch a page with full browser rendering:
import requests
import os
response = requests.post(
"https://api.fetch.tinyfish.ai",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"]},
json={"urls": ["https://example.com"], "format": "markdown"}
)
results = response.json().get("results", [])
print(results[0]["text"][:500] if results else "No content") # Clean markdown outputRun a goal-directed extraction:
response = requests.post(
"https://agent.tinyfish.ai/v1/automation/run",
headers={"X-API-Key": os.environ["TINYFISH_API_KEY"]},
json={
"url": "https://example.com/products",
"goal": "Extract all product names and prices. Return as JSON array with fields: name, price."
},
timeout=120
)
if not response.ok:
print(f"HTTP error: {response.status_code}")
else:
data = response.json()
# COMPLETED means the run finished — not necessarily that the goal succeeded
if data.get("status") == "COMPLETED" and data.get("result"):
print(data["result"]) # Structured JSON output
else:
print(f"Agent status: {data.get('status')} | error: {data.get('error')}")The API call pattern is close to a standard requests.post(). The structured output removes the parsing step for extraction tasks.
Yes. For static HTML sites, documentation, RSS feeds, and sites with public APIs, requests is still the fastest and simplest option. The cases where it's insufficient are specific: JavaScript-rendered content, strict automation requirements at scale, and multi-step authenticated workflows. If your current requests-based scripts are working, there's no reason to replace them.
Playwright is a browser automation library — you manage the browser process locally, write the code that controls it, and handle the infrastructure. TinyFish is a managed platform with a Browser API (CDP-compatible, your Playwright code connects to it) and a Web Agent layer (goal-directed, you describe what you want rather than how to get it). Playwright is better for local development, UI testing, and direct browser control. TinyFish is better when you want managed infrastructure, don't want to run browser servers, or need goal-based extraction.
TinyFish's Fetch API handles JS rendering remotely. Send a POST request to https://api.fetch.tinyfish.ai with "browser": true and your target URL. You get back clean markdown or HTML from the fully-rendered page — no browser process to manage, no chromedriver to maintain. It's a single API call that behaves like requests.get() but with a real browser behind it.
For basic rate limiting: add time.sleep() between requests (1–3 seconds per domain is a reasonable starting point). For more sophisticated limiting, use asyncio with semaphores to limit concurrent requests per domain. For production at scale, per-domain rate limiting with exponential backoff on 429 responses is necessary — the tenacity library handles this cleanly. TinyFish's API handles rate management for requests routed through it, so you only need to manage rate limits for direct requests you make yourself.
BeautifulSoup4 with the lxml parser is the standard choice — faster than the built-in parser and handles malformed HTML well. For very large HTML documents or high-performance parsing, lxml directly (without BS4) is faster. parsel (from the Scrapy project) has a cleaner XPath and CSS selector interface than BS4 for complex extraction tasks. html.parser from the standard library works for simple cases where you can't install external dependencies.
No credit card. No setup. Run your first operation in under a minute.
