Web agents also make sense when extraction accuracy has downstream consequences. MAP violation reports submitted with incorrect prices are worse than no report — they expose you to retailer disputes you can't back up. A extraction that silently returns the wrong price is a liability. An agent run with a failure status is just a gap in your data, which is recoverable.
You have 1,000 SKUs. Sold across 20 retailers. You need daily pricing data.
That's 20,000 data points to collect every morning — before your pricing team starts work.
Traditional scrapers were built for a simpler web. Static HTML, predictable selectors, no JavaScript. Today's retail sites are the opposite: prices rendered client-side, layout A/B tests that silently break your CSS selectors every few weeks, bot detection that blocks entire IP ranges. Maintaining that infrastructure isn't a one-time project. It becomes a full-time engineering responsibility.
The failure mode is quiet and expensive. Your extraction returns stale data with no error. A competitor drops below your Minimum Advertised Price (MAP) price. You find out three weeks later when a distributor emails to complain. By then, other retailers have matched the low price, and you're managing a channel conflict instead of preventing one.
Web agents change the architecture of the problem. Instead of maintaining brittle selectors across 20 retailer codebases, you describe the goal in plain English — "find the current price of this product" — and the agent navigates, renders, and extracts exactly as a human would. No selectors to maintain. No silent failures when layouts change.
This article walks through three production-ready use cases: competitor price monitoring, MAP violation detection, and dynamic pricing intelligence — with working code for each.
Before the use cases: if you're still deciding whether this applies to you, here's the honest decision framework.
The pattern: give the agent a list of product URLs, a structured output schema in the goal prompt, and run them concurrently. Total time equals the slowest single task — not the sum of all tasks.
pip install tinyfish
export TINYFISH_API_KEY=sk-tinyfish-*****import asyncio
import json
from datetime import datetime, timezone
from tinyfish import AsyncTinyFish, BrowserProfile
client = AsyncTinyFish() # Reads TINYFISH_API_KEY from environment
PRODUCTS = [
{"product_id": "airpods-pro-3", "url": "https://retailer-a.example.com/site/.."},
{"product_id": "airpods-pro-3", "url": "https://retailer-b.example.com/dp/.."},
{"product_id": "sony-wh1000xm6", "url": "https://retailer-c.example.com/p/.."},
# Add up to 1,000 product URLs
]
async def extract_price(product_id: str, url: str) -> dict:
goal = (
"Find the current listed price of the product on this page. "
"Return a JSON object with these fields only: "
"price (number, no currency symbol), currency (ISO 4217 code), in_stock (boolean). "
"If no price is visible, set price to null."
)
response = await client.agent.run(
url=url,
goal=goal,
browser_profile=BrowserProfile.MANAGED,
)
# For debugging: response.streaming_url contains a live browser replay of this run (valid 24h)
# response.result is a dict shaped by your goal prompt — or None if the run failed
# at the infrastructure level (browser crash, timeout, etc.)
result = response.result or {}
# Two distinct failure modes to handle:
# 1. Infrastructure failure: response.result is None → caught by `or {}`
# 2. Goal failure: run completed but agent couldn't achieve the task →
# result contains {"status": "failure", "reason": ".."} instead of your data
# This is the most common production surprise — COMPLETED status ≠ goal achieved
if result.get("status") "failure":
return {
"product_id": product_id,
"price": None,
"error": result.get("reason", "goal_failed"),
"scraped_at": datetime.now(timezone.utc).isoformat(),
"source_url": url,
}
# Happy path: result contains the fields you specified in the goal prompt
return {
"product_id": product_id,
"price": result.get("price"), # number, as specified in goal
"currency": result.get("currency", "USD"),
"in_stock": result.get("in_stock"), # boolean, as specified in goal
"scraped_at": datetime.now(timezone.utc).isoformat(),
"source_url": url,
}
async def main():
# All requests fire concurrently — total time = slowest single task
tasks = [extract_price(p["product_id"], p["url"]) for p in PRODUCTS]
results = await asyncio.gather(*tasks)
print(json.dumps(results, indent=2))
asyncio.run(main())Note on concurrency: When you send more requests than your plan's concurrent session limit, TinyFish queues the excess runs automatically (status: PENDING) — they start as sessions free up. Size your batches to your plan's concurrency limit for predictable run times.Note on result handling:status: "COMPLETED"means the browser ran successfully — not that your goal succeeded. A run that hit an Access Denied page will also returnCOMPLETED, butresultwill contain{"status": "failure", "reason": ".."}instead of your data. The code above handles both cases explicitly. This is the most common source of silent failures in production price monitors.
[
{
"product_id": "airpods-pro-3",
"price": 249.99,
"currency": "USD",
"in_stock": true,
"scraped_at": "2026-03-27T14:32:01Z",
"source_url": "https://retailer-a.example.com/site/.."
},
{
"product_id": "airpods-pro-3",
"price": 239.00,
"currency": "USD",
"in_stock": true,
"scraped_at": "2026-03-27T14:32:04Z",
"source_url": "https://retailer-b.example.com/dp/.."
}
]| Approach | 1,000 products @ 3s each | Actual wall-clock time |
|---|---|---|
| Sequential extraction | 1,000 × 3s | ~50 minutes |
| TinyFish concurrent agents | 1,000 × 3s | ~3–5 minutes |
The math: total time equals the slowest single task, not the sum. At 1,000 concurrent agents, a batch that took 50 minutes sequentially completes in the time it takes to process one page.
| Retailer category | Success rate |
|---|---|
| Major global e-commerce platforms | up to 95–100% (e-commerce category; internal testing) |
| Sites with enterprise bot management | Lower — see anti-bot section |
|---|
MAP enforcement has a discovery problem. Your team can't manually check every authorized retailer every day — and by the time a violation surfaces through a distributor complaint, the channel damage is done.
A scheduled agent run closes the gap: extract prices, compare against your MAP database, generate a timestamped evidence record per violation. The whole workflow runs while your team sleeps.
import asyncio
import json
from datetime import datetime, timezone
from tinyfish import AsyncTinyFish, BrowserProfile
client = AsyncTinyFish()
# Your MAP pricing database
MAP_PRICES = {
"airpods-pro-3": 249.00,
"sony-wh1000xm6": 299.00,
}
# Authorized retailers to monitor per product
RETAILERS = {
"airpods-pro-3": [
{"name": "BestBuy", "url": "https://retailer-a.example.com/site/.."},
{"name": "Target", "url": "https://retailer-c.example.com/p/.."},
{"name": "Walmart", "url": "https://retailer-d.example.com/ip/.."},
],
}
async def check_retailer(product_id: str, map_price: float, retailer: dict) -> dict | None:
response = await client.agent.run(
url=retailer["url"],
goal=(
"Return a JSON object with: price (number, no symbol), currency (ISO 4217). "
"If no price is found, set price to null."
),
browser_profile=BrowserProfile.MANAGED,
)
result = response.result or {}
advertised = result.get("price")
if advertised is None or result.get("status") "failure":
return None # Could not extract price — log separately
if advertised < map_price:
return {
"product_id": product_id,
"retailer": retailer["name"],
"map_price": map_price,
"advertised_price": advertised,
"violation_amount": round(map_price - advertised, 2),
"evidence_url": retailer["url"],
"detected_at": datetime.now(timezone.utc).isoformat(),
}
return None # Compliant
async def run_map_check():
all_tasks = []
for product_id, map_price in MAP_PRICES.items():
for retailer in RETAILERS.get(product_id, []):
all_tasks.append(check_retailer(product_id, map_price, retailer))
results = await asyncio.gather(*all_tasks)
violations = [r for r in results if r is not None]
print(json.dumps(violations, indent=2))
return violations
asyncio.run(run_map_check()){
"product_id": "airpods-pro-3",
"retailer": "Target",
"map_price": 249.00,
"advertised_price": 219.99,
"violation_amount": 29.01,
"evidence_url": "https://retailer-c.example.com/p/..",
"detected_at": "2026-03-27T14:35:22Z"
}This structure pipes directly into your brand protection workflow. Route violations to Slack for same-day retailer outreach, to your ERP for automated distributor notification, or to a compliance dashboard. The timestamped URL is your legal evidence record — captured at the exact moment of detection.
Static MAP enforcement catches violations after the fact. The harder problem is building a pricing system that reacts to market changes before they compound.
The structural issue: prices on food delivery, travel, and marketplace platforms load dynamically per user session, vary by location, and change multiple times per day. A extraction built on CSS selectors breaks every time a vendor updates their layout — and in high-velocity markets, that's often. An agent built on a goal — "find the current price of this item at this location" — adapts to layout changes automatically, because it's reading the page the way a human would, not pattern-matching against a selector you wrote last quarter.
The output that makes this useful downstream isn't a report — it's a structured feed with enough granularity to drive pricing decisions:
{
"market_id": "sf-94102",
"restaurant_id": "R_8821",
"item_id": "burger-classic",
"base_price": 12.99,
"delivery_fee": 1.99,
"promo_price": null,
"scraped_at": "2026-03-27T18:00:00Z"
}The code structure to produce this is identical to Use case 1 — AsyncTinyFish + asyncio.gather() across a list of URLs, with a goal prompt that specifies the schema above. The only difference is the schema itself. If you've already built Use case 1, this is a goal prompt change, not an architecture change.
A major food delivery platform uses this pattern to track millions of pricing variables per month across thousands of markets. The scale is unusual; the architecture isn't.
TinyFish achieves up to 85% success rate on sites with strict automation requirements. In practice, this covers the majority of consumer retail platforms most teams monitor.
The exception is sites running enterprise behavioral analysis systems. These systems don't look for a missing header — they model whether the entire session pattern looks human. Success rates are lower and inconsistent across all automation tools. No vendor publishes reliable numbers for the hardest-protected sites, and you should be skeptical of any that do.
Practical approach for protected retailers:
from tinyfish import TinyFish, BrowserProfile, ProxyConfig, ProxyCountryCode
client = TinyFish()
result = None
with client.agent.stream(
url="https://protected-retailer.example.com/product/..",
goal="Extract the current price and stock status",
browser_profile=BrowserProfile.MANAGED,
proxy_config=ProxyConfig(enabled=True, country_code=ProxyCountryCode.US),
) as stream:
for event in stream:
# SDK CompleteEvent: result lives in event.result_json
if getattr(event, "type", None) "COMPLETE":
# Layer 1: infrastructure failure
result = event.result_json
break
# Layer 2: goal failure
if result and isinstance(result, dict) and result.get("status") "failure":
result = NoneTinyFish handles detection at the infrastructure level — not through JS injection applied after browser start. Adding a matching country-code proxy handles geo-specific access requirements. For sites that still reliably block: weigh whether the data value justifies ongoing engineering time, or whether an alternative data source exists.
TinyFish pricing by plan: Pay-as-you-go $0.015/step · Starter $15/mo (1,650 included steps, $0.014/step overage) · Pro $150/mo (16,500 included steps, $0.012/step overage) · Enterprise custom. One price check = one step.
Free on all plans (no credits consumed): Search API and Fetch API — use these for initial product URL discovery and for retailers whose pages don't require full browser execution. Credits apply only to Web Agent steps.
| Scale | Daily steps | Monthly steps | Pay-as-you-go | Pro plan |
|---|---|---|---|---|
| 100 products × 5 retailers | 500 | ~15,000 | ~$225/mo | $150/mo (within 16,500 included) |
| 500 products × 10 retailers | 5,000 | ~150,000 | ~$2,250/mo | $150 + ~$1,602 overage ≈ $1,752/mo |
| 1,000 products × 20 retailers | 20,000 | ~600,000 | ~$9,000/mo | $150 + ~$7,002 overage ≈ $7,152/mo |
| Enterprise scale | millions/month | — | — | Enterprise |
The free tier — 500 steps, no credit card — covers a complete scan of up to 500 product URLs. Enough to test your real target retailer list and validate extraction quality before committing to a production schedule.
For context on extraction maintenance costs: a junior engineer spending 25% of their time keeping selectors current across 20 retailer sites is a real line item. That's not in the table above.
The honest answer is that many teams don't need a web agent for price monitoring. If your requirements are under 50 products, static pages, and a handful of stable low-protection retailers — Scrapy or a basic Playwright script is cheaper. Build it yourself.
The crossover point isn't a product decision, it's a maintenance math question: when does the engineering time spent keeping selectors current across 20 retailer sites exceed the cost of outsourcing that infrastructure? That typically happens around 100+ products, 5+ retailers, or the first time a retailer redesign takes down your monitoring for a week without anyone noticing.
Web agents also make sense when extraction accuracy has downstream consequences. MAP violation reports submitted with incorrect prices are worse than no report — they expose you to retailer disputes you can't back up. A extraction that silently returns the wrong price is a liability. An agent run with a failure status is just a gap in your data, which is recoverable.
The free tier gives you 500 steps with no credit card required — enough to run a complete price scan across 500 products and validate results against your real target retailers.
[Start free → agent.tinyfish.ai]
For teams monitoring 10,000+ products or needing SLA guarantees, [contact enterprise sales] for volume pricing and dedicated support.
Can web agents handle price monitoring on major e-commerce platforms?
Yes. Major e-commerce platforms run at up to 95–100% success rate (e-commerce category; internal testing). Standard browser profile covers most product pages; switch to stealth mode for high-volume or well-protected retailers.
How fresh is the price data?
Agents run on demand, so freshness is whatever schedule you set. Daily is the most common pattern. For flash-sale categories, teams commonly run 4× daily or trigger runs on inventory alerts.
Does this work for international retailers?
Yes. Include a currency field (ISO 4217) in your goal schema. For regional price variants — products that display different prices by country — include proxy_config with the matching country code (US, GB, CA, DE, FR, JP, AU supported) to retrieve the correct regional pricing.
What does `COMPLETED` status actually mean?
Infrastructure success only — the browser launched and finished. It does not mean your goal succeeded. Always check the result field: if it contains {"status": "failure"}, the agent ran but couldn't extract the data. This is the most common production gotcha.
What if one agent in a batch fails?
Each run is independent — one failure doesn't affect others. Every run includes a streaming_url for debugging (valid 24 hours). Failed runs where infrastructure succeeded are not billed.
Is there a concurrency limit?
Yes. Exceeding it triggers automatic queuing — no 429 errors, but later requests take longer. Check your plan's limit in the dashboard.
No credit card. No setup. Run your first operation in under a minute.
