November 10, 2025
Industry

The Numbers That Don't Exist

Enterprises have been making billion-dollar decisions blind—not from laziness, but because gathering the data was mathematically impossible until now.
Hidden Web
Web Agents
Industry
November 10, 2025
Industry

The Numbers That Don't Exist

AI summary by TinyFish
  • A retailer needed to track 50,000 product pages daily to understand pricing—but the work was too large to ever attempt.
  • The web resists automation: authentication changes, product mismatches, and fragmented data make monitoring unreliable for human teams.
  • Even with AI, validation and constant site changes make the task unscalable.
  • As prices shift by the minute, data gathered manually is outdated before it’s complete.
  • TinyFish’s infrastructure makes this once-impossible work feasible—turning blind spots into operational intelligence.
  • A pricing team at a mid-sized retailer wanted to understand their competitive position across their catalog. Reasonable question. The answer required checking 50,000 product pages daily. At 30 seconds per page—if you could somehow maintain that pace without breaks—that's 417 hours of work. Every single day.

    They never tried. The work remained undone because it couldn't be scoped.

    As Box CEO Aaron Levie recently noted:

    “some of the most interesting use-cases that keep coming up for AI agents are on bringing automated work to areas that the companies would not have been able to apply labor to before.”

    We're not talking about efficiency gains. We're talking about work that enterprises never attempted because the economics were impossible.

    When the Web Resists at Scale

    At TinyFish, we build enterprise web agent infrastructure—systems that handle reliable automation across thousands of sites simultaneously. Building this infrastructure shows you exactly why certain monitoring tasks remained undone for human teams.

    The web actively fights back. Authentication flows change without warning. A retailer checking competitor pricing finds that a site working perfectly yesterday now requires two-factor verification. Or the login page restructures. Or regional variations mean the flow works in the US but breaks in Japan. Each site has dozens of these edge cases.

    Product matching creates its own nightmare. The same item appears with different descriptions, images, and naming conventions across hundreds of stores. Even AI-driven matching starts at 80-90% accuracy, requiring human validation to approach 100%. Multiply that across thousands of SKUs and the validation work alone becomes impossible to resource.

    The data fragments. Price lives in one platform, sentiment in another, stock levels buried in outdated reports. You need to pull from desktop sites, mobile sites, apps—each requiring different technical approaches. The work isn't just large. It's architecturally impossible for human teams.

    But the temporal dimension breaks everything. Amazon adjusts prices every few minutes. Even if you could check multiple times daily, gathering information from all vendors will take months. By the time you finish, the information you collected three months ago has changed. You're always working with stale data because the collection cycle never completes.

    So retailers made pricing decisions blind. They ran promotions without competitive context. They managed inventory based on quarterly snapshots. Not because they didn't understand the value of real-time data, but because comprehensive monitoring "can seem impossible".

    What Was Lost

    When enterprises realize they could actually know things they've been operating blind on, something shifts. The conversation stops being about automation capabilities and becomes about recognition: "Wait, we've been making decisions without this information?"

    The work that stayed undone:

    • Pricing strategies built on guesswork
    • Inventory decisions based on months-old data
    • Promotional timing that missed competitive windows

    The work existed. The information was theoretically available. But until infrastructure existed to make it economically viable, certain questions simply couldn't be answered. The numbers never added up.

    Share article
    Rina Takahashi