Open-Source Web Scraping Tools: A Hands-On Comparison for Developers

Want rendered pages without running browsers yourself? Try TinyFetch free. TinySearch and TinyFetch are free within their limits.

TL;DR

Most lists of open source web scraping tools rank twenty libraries without running any of them. I took three approaches that cover the three jobs in scraping, ran them on the same two public test pages, and measured what came back:

• An HTTP client and a parser (requests + BeautifulSoup) was the lightest and simplest: 100 of 100 quotes from the static page in 6.5 seconds, using about 48 MB of memory. On the JavaScript version of the page it returned 0 quotes, with no error.
• A crawling framework (Scrapy) also got 100 of 100 from the static page, and gave me robots.txt checks, throttling, retries and JSON export for free. With its AutoThrottle setting on, the same crawl took 20 seconds instead of 6. It also returned 0 quotes from the JavaScript page.
• A browser (Playwright) was the only one that got 100 of 100 from both pages. It cost a 100 MB browser download and about 550 MB of memory while running, more than 10 times the parser.

The lesson isn't that one tool wins. Parsing, crawling and rendering are different jobs. Use the lightest one that gets the data, and add the others only when the page or the scale needs them.

A quick disclosure: I work at TinyFish. The comparison below uses only open-source tools. I add one short section at the end on where a managed fetch API like TinyFetch fits, with the same test.

The test setup

To keep this honest, every tool ran the same task on the same disclosed fixtures:

• Static fixture: quotes.toscrape.com, a public sandbox built for scraping practice. 10 pages, 10 quotes each, with a "Next" link.
• JavaScript fixture: quotes.toscrape.com/js/, the same 100 quotes, but the page builds them in the browser with JavaScript. The HTML that arrives over the network contains no quote elements.
• Task: collect every quote's text, author and tags across all 10 pages, following the "Next" link.
• Politeness: every tool waits 0.5 seconds between pages and sends a clear user agent. The site serves no robots.txt (it returns 404), so nothing is disallowed.
• Machine: an Apple M2 Max laptop on a home connection, Python 3.12.15, run on November 9, 2026, at about 7:40 PM PT.
• Versions: requests 2.34.2, BeautifulSoup 4.15.0, Scrapy 2.20.0, Playwright 1.65.0 with Chrome Headless Shell 158.

Times include the 0.5-second waits, so they compare the tools fairly with each other, not with your production numbers. Memory is the peak resident memory of the whole process tree, including any browser processes.

Parsing, crawling and rendering are different jobs

Before the code, it helps to separate three jobs that "web scraping tool" lumps together:

• Parsing turns HTML you already have into data. CSS selectors or XPath find the elements, and you read their text and attributes. BeautifulSoup, lxml and selectolax are parsers.
• Crawling decides which pages to fetch and fetches them at scale: a queue of URLs, a scheduler, politeness rules, retries, deduplication and exporting the results. Scrapy is a crawling framework. It still needs a parser inside, and it uses its own.
• Rendering runs the page's JavaScript in a real browser, so content built on the client side exists before you read it. Playwright, Puppeteer and Selenium render.

Most real scrapers combine them: crawl to find pages, render only the pages that need it, and parse every time. The mistake I see most is reaching for the heaviest job first.

Approach 1: HTTP client and parser (requests + BeautifulSoup)

Setup:

pip install requests beautifulsoup4

Code (27 lines without blanks and comments):

import sys, time, requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

def scrape(start_url):
    session = requests.Session()
    session.headers["User-Agent"] = "tinyfish-blog-demo/1.0"
    url, quotes = start_url, []
    while url:
        resp = session.get(url, timeout=10)
        resp.raise_for_status()
        soup = BeautifulSoup(resp.text, "html.parser")
        for q in soup.select("div.quote"):
            quotes.append({
                "text": q.select_one("span.text").get_text(strip=True),
                "author": q.select_one("small.author").get_text(strip=True),
                "tags": [t.get_text(strip=True) for t in q.select("a.tag")],
            })
        nxt = soup.select_one("li.next > a")
        url = urljoin(url, nxt["href"]) if nxt else None
        time.sleep(0.5)  # be polite
    return quotes

if __name__ == "__main__":
    t0 = time.perf_counter()
    quotes = scrape(sys.argv[1])
    print(f"{len(quotes)} quotes in {time.perf_counter() - t0:.1f}s")
    for q in quotes[:2]:
        print(q)

What came back from the static page:

100 quotes in 6.5s
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}

What came back from the JavaScript page:

0 quotes in 6.5s

Notice that the JavaScript run didn't fail. It followed the "Next" link through all 10 pages, because the pagination links are in the HTML, and found zero quotes on each one, because the quotes aren't. A scraper that returns an empty list with no error is the most dangerous failure in this whole post. Always check that a run returned what you expected.

Measured: 6.5 seconds, about 48 MB peak memory, 100 of 100 quotes on the static page, 0 of 100 on the JavaScript page.

The work it leaves to you:

• Pagination, retries, timeouts and backoff are your code. Here they're a while loop and raise_for_status, which is fine for 10 pages and fragile for 10,000.
• robots.txt is your job too. requests doesn't read it. Python's urllib.robotparser can.
• When the site changes its class names, your selectors break silently. Add a check like "at least one quote per page".
• A trick worth knowing: the JavaScript page carries its data in a var data = [...] block inside a script tag. I pulled it out with one regex and json.loads, and got all 10 quotes on the first page, including tags. Reading embedded JSON or a site's own API is often lighter than rendering. It's also more brittle, since the site can change that block at any time.

Approach 2: crawling framework (Scrapy)

Setup:

pip install scrapy
scrapy runspider quotes_spider.py -O quotes.json

Code (23 lines without blanks and comments):

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "DOWNLOAD_DELAY": 0.5,
        "AUTOTHROTTLE_ENABLED": True,
        "USER_AGENT": "tinyfish-blog-demo/1.0",
        "LOG_LEVEL": "INFO",
    }

    def __init__(self, start="https://quotes.toscrape.com/", **kwargs):
        super().__init__(**kwargs)
        self.start_urls = [start]

    def parse(self, response):
        for q in response.css("div.quote"):
            yield {
                "text": q.css("span.text::text").get(),
                "author": q.css("small.author::text").get(),
                "tags": q.css("a.tag::text").getall(),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

What came back: 100 items in quotes.json for the static page, the same first quote as above. Scrapy's own stats from the run:

'downloader/request_count': 11,
'downloader/response_status_count/200': 10,
'item_scraped_count': 100,
'robotstxt/request_count': 1,
'robotstxt/response_status_count/404': 1,
'elapsed_time_seconds': 20.12749108299613,

The extra request is robots.txt: Scrapy checked it before crawling because I set ROBOTSTXT_OBEY. On the JavaScript page it made the same 11 requests and exported 0 items.

Measured: 20.1 seconds with AutoThrottle on, 6.0 seconds with it off, about 85 to 95 MB peak memory, 100 of 100 on the static page, 0 of 100 on the JavaScript page.

The 20 seconds isn't Scrapy being slow. AutoThrottle starts cautiously and adjusts its delay to how fast the server responds, which is the polite default you want on a real site. With it off, Scrapy matched the hand-written loop.

The work it takes off your plate: robots.txt, throttling, concurrency, retries, duplicate filtering, export to JSON or CSV, and logging with statistics. That's most of what the first approach left to you.

The work it adds: learning its model (spiders, callbacks, settings, middlewares, pipelines), and a bigger process. It still doesn't run JavaScript. For that you plug in a renderer, for example the scrapy-playwright package, and inherit the browser costs below.

Approach 3: browser automation (Playwright)

Setup:

pip install playwright
playwright install chromium   # downloaded 100.1 MiB here; about 205 MB on disk

Code (30 lines without blanks and comments):

import sys, time
from playwright.sync_api import sync_playwright

def scrape(start_url):
    quotes = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(user_agent="tinyfish-blog-demo/1.0")
        page.goto(start_url, timeout=30_000)
        while True:
            page.wait_for_selector("div.quote", timeout=10_000)
            for q in page.locator("div.quote").all():
                quotes.append({
                    "text": q.locator("span.text").inner_text(),
                    "author": q.locator("small.author").inner_text(),
                    "tags": q.locator("a.tag").all_inner_texts(),
                })
            nxt = page.locator("li.next > a")
            if nxt.count() == 0:
                break
            nxt.click()
            page.wait_for_load_state("domcontentloaded")
            time.sleep(0.5)
        browser.close()
    return quotes

if __name__ == "__main__":
    t0 = time.perf_counter()
    quotes = scrape(sys.argv[1])
    print(f"{len(quotes)} quotes in {time.perf_counter() - t0:.1f}s")
    for q in quotes[:2]:
        print(q)

What came back from the JavaScript page:

100 quotes in 9.3s
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}

The static page gave the same 100 quotes.

Measured: 9.3 seconds on the JavaScript page and 13.7 seconds on the static page (plus about a second to launch the browser), with about 549 to 566 MB peak memory across the browser processes. 100 of 100 on both.

The work it adds:

• Browser binaries. Every machine, container and CI runner needs the right browser build, and it has to be updated with Playwright.
• Memory. Over 10 times the parser here, for one tab. Running many pages at once multiplies it.
• Waits. You have to wait for the right thing (wait_for_selector here) instead of a fixed sleep, or runs become flaky.
• Headless differences. Some sites behave differently in headless browsers, and a few block them outright.
• Scaling. Ten concurrent browsers is a server-sizing problem, not a loop.

Results side by side

All three were short. The real differences are in the last four rows: what each tool asks you to install, run and maintain.

Maintenance: what breaks in production

The test pages never change. Real sites do, and that's where most scraping time goes:

• Selectors break. A redesign renames span.text, and every tool here returns empty results until you fix it. Monitor counts per page, not just errors.
• Pagination changes. "Next" links become infinite scroll or a "Load more" button. The parser and Scrapy then need the site's API. The browser needs clicking or scrolling logic.
• Blocking and rate limits. Throttle, back off on 429 and 503 responses, and identify yourself. Scrapy gives you most of this. With the other two you write it.
• Rules. Read robots.txt and the site's terms before you scrape, and don't collect personal data you don't need. Practice sites like these exist so you can learn without bothering anyone.
• Runtime. Parsers run anywhere. Browsers need memory, binaries and patience at scale, which is exactly why hosted browsers exist. Our cloud browser comparison covers that side.

Which one should you start with?

• The page has the data in its HTML: start with an HTTP client and a parser. It's the fastest to write and the cheapest to run.
• You need many pages, politely, on a schedule: use Scrapy, and keep its throttling on.
• The data only appears after JavaScript runs: first look for an API or embedded JSON. Only then add a browser, and only for the pages that need it.

For the longer version of that choice, with timing on 30 real URLs, read web scraping with requests vs a headless browser. For a Python walkthrough from scratch, see web scraping in Python.

Rendering JavaScript pages without running a browser fleet? Get a free TinyFish API key and try TinyFetch on your own pages.

Where a managed fetch fits

The open-source tools above are the right start for most projects. The point where teams look elsewhere is usually rendering at scale: keeping browsers patched, sized and unblocked.

That's the job TinyFetch does. I ran the same JavaScript fixture through it, asking for HTML scoped to div.quote. It came back rendered, with the same markup Playwright saw, in about 2.7 seconds for the page:

<div class="quote"><span class="text">“The world as we have created it is a process of our thinking. ...”</span>
<span>by <small class="author">Albert Einstein</small></span>
<div class="tags">Tags: <a class="tag">change</a> <a class="tag">deep-thoughts</a> ...</div></div>

So the parser from the first approach works on it unchanged, with no browser on your machine. To be clear about the limits: you still own pagination and selectors, and fetching a page is not the same as validating what's on it. TinyFetch is free within its limits, and it sits behind the same API as TinySearch for finding pages and TinyBrowser for when you do need to drive a browser yourself.

Further reading

This post compares approaches. For the full market, including hosted APIs, no-code tools and AI extractors, see our buying guide to the best web scraping tools. Related reading:

• Best structured data extraction tools
• Production-grade web fetching for AI agents
• Web scraping project ideas with source code

Ready to skip the browser setup? Start free with TinyFish: TinySearch and TinyFetch are free within their limits, and new accounts get $8 for TinyAgent and TinyBrowser.