How to Monitor 1,000 Websites in Parallel with the TinyFish API
Your monitoring setup catches downtime. It doesn't catch the competitor who quietly dropped their price by 15%, the supplier whose inventory hit zero three days ago, or the regulatory page that updated its requirements last Tuesday.
Uptime monitoring and content monitoring are different problems. The first is solved. The second — monitoring what's on pages at scale, across authenticated portals, with structured output — is still largely manual or brittle.
At 10 sites, a cron job and a headless browser works fine. At 100, it starts to creak. At 1,000, you're managing a fleet of browser processes, rotating proxies, handling session state, and debugging access failures — before you've written a single line of business logic.
TinyFish's agent API is built around a different model: describe what to check, run it in parallel, get structured JSON back. Here's how to build a monitoring pipeline that scales to 1,000 sites without managing the infrastructure.
What Traditional Monitoring Misses
Standard monitoring tools — Uptime Robot, Pingdom, Better Uptime — check whether a URL returns a 200. That's useful for availability. It tells you nothing about:
- Price changes across competitor product pages
- Inventory status on supplier portals (which require login)
- Regulatory updates on government or compliance sites
- Content drift — when a competitor's positioning or messaging changes
- Feature launches tracked through changelog pages
For these use cases, you need a monitor that reads and understands the page, not just pings it.
The TinyFish Monitoring Architecture
TinyFish's Web Agent API accepts a natural language goal and returns structured JSON. For monitoring, the pattern is:
- Define what to check per site (the goal)
- Batch sites into parallel groups
- Collect results and compare against previous state
- Alert on diffs
With the default TinyFish account limit of two concurrent Agent runs, you can check two sites at once. For 1,000 sites, that means 500 waves of two runs. To run 50 sites simultaneously, arrange an enterprise concurrency limit with TinyFish; at 50 concurrent runs, 1,000 sites become 20 waves of 50.
A wave finishes when its slowest task finishes. TinyFish estimates 3–10 seconds for simple pages and 30–60 seconds for complex multi-step automations, but actual runtime also depends on page load times, authentication, anti-bot handling, retries, and the goal itself.
Setting Up Your First Monitoring Job
Authentication
All API calls use an API key from your TinyFish dashboard, passed via the X-API-Key header:
Single Site Check
Before batching, test your goal description against one site:
Batching 1,000 Sites
Set your application’s concurrency to your account’s current Agent-run limit. Self-service starts at two concurrent runs; enterprise concurrency is configured for the workload:
For a small set of immediate results, use concurrent requests with asyncio.gather() or Promise.all(). For larger or long-running batches, use agent.queue() in the SDK, POST /v1/automation/run-batch in REST, or tinyfish agent batch run in the CLI. Validate each run’s result, not only its COMPLETED status, because a browser run can complete without achieving the extraction goal.
Handling Results and Diffs
Monitoring Authenticated Portals
For sites that require login, include credentials in your goal description. TinyFish agents handle authentication as part of the task:
The safety instruction (Do not proceed to any checkout or payment flow) matters for any commercial workflow — it prevents the agent from accidentally triggering transactions.
Monitoring at Different Frequencies
Not all sites need the same cadence. Group by volatility:
Cost Estimation
Cost depends on whether a monitoring task uses only free retrieval or needs an Agent or Browser session.
Search and Fetch are free at every Wallet balance: Search is limited to 30 requests per minute and Fetch to 150 URLs per minute. Neither product draws from the Wallet. Remove the statement about failed fetches being uncharged—the current pricing page does not publish separate failure accounting because Fetch itself is free.
TinyFish no longer has a Pro plan, included steps, or monthly overage rates. Agent usage costs $0.016 per step and Browser usage costs $0.002 per minute, both deducted from prepaid Wallet funds. Accounts start with two concurrent Agent runs and five concurrent Browser sessions; higher limits are set with sales for enterprise workloads.
For example, a five-step Agent monitoring check costs about $0.08, while a 15-step check costs about $0.24. For 3,300 checks, that would be roughly $264–$792 in Agent-step usage, depending on task complexity. Browser time, if used separately, is additional.
To reduce cost, keep the goal narrow. “Find the price of the main product” will often need fewer Agent steps than “Find all prices, discounts, bulk tiers, and promotions,” though the exact step count depends on the site and interactions required.
What This Handles That Cron + Playwright Doesn't
Sites with strict automation requirements. Sites with strict automation requirements work with TinyFish's infrastructure-level handling — no managing puppeteer-extra plugins or proxy rotation per site.
Layout changes. When a competitor redesigns their pricing page and your CSS selectors break, the agent reads the page and finds the price regardless of class names. No code update needed.
Authenticated portals at scale. Use Vault credentials or saved Browser Context Profiles for authenticated runs instead of placing passwords in the goal. Self-service accounts start with two concurrent Agent runs and five concurrent Browser sessions, new accounts get $8 in credit as part of our funding for your test usage. Noting that running 50 simultaneous authenticated workflows requires pacing per minute; it is not automatically enabled by sending 50 API calls.
Structured output. Instead of HTML to parse, you get JSON with the fields you requested. No BeautifulSoup, no regex, no post-processing pipeline.
TinyFish gives you $8 in wallet credit to test against your actual monitoring targets — no credit card required.
FAQ
How many sites can I monitor in parallel?
Up to your account’s concurrent Agent-run limit: self-service accounts start at two concurrent runs. TinyFish does not currently use Starter or Pro plans; higher concurrency is configured with TinyFish for enterprise workloads.
For 1,000 sites at the default limit, run 500 waves of two. If your enterprise account is configured for 50 concurrent runs, run 20 waves of 50. Each wave takes roughly as long as its slowest task. TinyFish estimates 3–10 seconds for simple pages and 30–60 seconds for complex multi-step automations, though authentication, site speed, anti-bot handling, and retries can extend that time.
How does TinyFish handle sites that block bots?
TinyFish runs a native Chromium-based browser session with infrastructure-level request handling. Residential proxy routing is included on all plans at no extra cost. Success rates vary by site complexity; enterprise-grade protection systems may need additional configuration.
Can I monitor sites that require login?
Yes. Include credentials in your goal description; the agent handles the authentication flow. Store credentials as environment variables, not in the goal string directly. Always add a safety instruction ("Do not proceed to checkout") for any commercial workflow.
What happens if a site changes its layout?
Nothing breaks on your end. The agent reads the page and extracts the data based on your goal description, not CSS selectors. Layout changes are invisible to your monitoring pipeline.
How do I send alerts when something changes?
Pipe the changes list from the diff function to your alerting stack — Slack via webhook, PagerDuty, email, or any webhook endpoint. TinyFish returns JSON; routing it to alerts is standard integration work.
Is this cheaper than a self-managed Playwright cluster?
At small scale (under 100 sites/day), self-managed Playwright on cheap compute is cheaper. At 100–1,000 sites/day, TinyFish's all-in pricing — browsers, proxies, LLM inference included — often undercuts the real cost of managing the infrastructure, especially accounting for engineering time on proxy rotation, anti-detection, and session management.
Try TinyFish Free
$8 in wallet credit allowance, no credit card. The fastest way to test whether TinyFish fits your workflow.
Related Reading
AI disclosure
Content on this website may be created or refined with the assistance of AI tools and is subject to human editorial review.
