How to Monitor 1,000 Websites in Parallel with the TinyFish API
Your monitoring setup catches downtime. It doesn't catch the competitor who quietly dropped their price by 15%, the supplier whose inventory hit zero three days ago, or the regulatory page that updated its requirements last Tuesday.
Uptime monitoring and content monitoring are different problems. The first is solved. The second — monitoring what's on pages at scale, across authenticated portals, with structured output — is still largely manual or brittle.
At 10 sites, a cron job and a headless browser works fine. At 100, it starts to creak. At 1,000, you're managing a fleet of browser processes, rotating proxies, handling session state, and debugging access failures — before you've written a single line of business logic.
TinyFish's agent API is built around a different model: describe what to check, run it in parallel, get structured JSON back. Here's how to build a monitoring pipeline that scales to 1,000 sites without managing the infrastructure.
What Traditional Monitoring Misses
Standard monitoring tools — Uptime Robot, Pingdom, Better Uptime — check whether a URL returns a 200. That's useful for availability. It tells you nothing about:
- Price changes across competitor product pages
- Inventory status on supplier portals (which require login)
- Regulatory updates on government or compliance sites
- Content drift — when a competitor's positioning or messaging changes
- Feature launches tracked through changelog pages
For these use cases, you need a monitor that reads and understands the page, not just pings it.
The TinyFish Monitoring Architecture
TinyFish's Web Agent API accepts a natural language goal and returns structured JSON. For monitoring, the pattern is:
- Define what to check per site (the goal)
- Batch sites into parallel groups
- Collect results and compare against previous state
- Alert on diffs
With 50 concurrent agents on the Pro plan, you can check 50 sites simultaneously. For 1,000 sites, that's 20 sequential batches — total runtime depends on per-site task complexity, typically 30 seconds to 3 minutes per batch.
Setting Up Your First Monitoring Job
Authentication
All API calls use an API key from your TinyFish dashboard, passed via the X-API-Key header:
Single Site Check
Before batching, test your goal description against one site:
Batching 1,000 Sites
With asyncio and aiohttp, run up to 50 concurrent requests matching your plan's concurrency limit:
Handling Results and Diffs
Monitoring Authenticated Portals
For sites that require login, include credentials in your goal description. TinyFish agents handle authentication as part of the task:
The safety instruction (Do not proceed to any checkout or payment flow) matters for any commercial workflow — it prevents the agent from accidentally triggering transactions.
Monitoring at Different Frequencies
Not all sites need the same cadence. Group by volatility:
Cost Estimation
Cost depends on how many steps each monitoring task takes:
Search and Fetch are free on all plans — rate-limited by plan tier (Free: 5 searches/min, 25 fetches/min). Failed fetches are never charged. Failed fetches are never charged. TinyFish runs its own browser infrastructure so there's no markup to pass on; failed fetches don't count against your quota either. Credits apply only to agent steps and browser sessions, as shown:
Pro plan (50 concurrent, $150/month, 16,500 steps included) covers ~3,300 simple site checks before overage. Beyond that, steps bill at $0.012.
To reduce cost: keep goal descriptions narrow. "Find the price of the main product" uses fewer steps than "Find all pricing including discounts, bulk tiers, and promotions."
What This Handles That Cron + Playwright Doesn't
Sites with strict automation requirements. Sites with strict automation requirements work with TinyFish's infrastructure-level handling — no managing puppeteer-extra plugins or proxy rotation per site.
Layout changes. When a competitor redesigns their pricing page and your CSS selectors break, the agent reads the page and finds the price regardless of class names. No code update needed.
Authenticated portals at scale. Running 50 simultaneous authenticated sessions across different supplier portals would require significant session management infrastructure. With TinyFish, it's 50 API calls with different credentials in the goal.
Structured output. Instead of HTML to parse, you get JSON with the fields you requested. No BeautifulSoup, no regex, no post-processing pipeline.
TinyFish gives you 500 free steps to test against your actual monitoring targets — no credit card required.
FAQ
How many sites can I monitor in parallel?
Up to your plan's concurrent agent limit: 2 on PAYG, 10 on Starter, 50 on Pro. For 1,000 sites on Pro, you run 20 sequential batches of 50. Runtime per batch depends on task complexity — simple checks: 10–30 seconds each; authenticated multi-step tasks: 1–3 minutes.
How does TinyFish handle sites that block bots?
TinyFish runs a native Chromium-based browser session with infrastructure-level request handling. Residential proxy routing is included on all plans at no extra cost. Success rates vary by site complexity; enterprise-grade protection systems may need additional configuration.
Can I monitor sites that require login?
Yes. Include credentials in your goal description; the agent handles the authentication flow. Store credentials as environment variables, not in the goal string directly. Always add a safety instruction ("Do not proceed to checkout") for any commercial workflow.
What happens if a site changes its layout?
Nothing breaks on your end. The agent reads the page and extracts the data based on your goal description, not CSS selectors. Layout changes are invisible to your monitoring pipeline.
How do I send alerts when something changes?
Pipe the changes list from the diff function to your alerting stack — Slack via webhook, PagerDuty, email, or any webhook endpoint. TinyFish returns JSON; routing it to alerts is standard integration work.
Is this cheaper than a self-managed Playwright cluster?
At small scale (under 100 sites/day), self-managed Playwright on cheap compute is cheaper. At 100–1,000 sites/day, TinyFish's all-in pricing — browsers, proxies, LLM inference included — often undercuts the real cost of managing the infrastructure, especially accounting for engineering time on proxy rotation, anti-detection, and session management.
Try TinyFish Free
500 free steps, no credit card. The fastest way to test whether TinyFish fits your workflow.
