How BigSet Works Under-the-Hood: Any Dataset, From a Sentence

It's 2026. Your agents can reason, plan, and act. They're still starving for data.

Which is strange, because the data is right there on the web: prices, companies, jobs, research, who's hiring, what's in stock, etc.

The problem is what you get when you go looking: the answer comes back in pieces. A few results here, a page there, half of them out of date, none of them lining up. Turning that into something you'd actually use, where every option side by side, the numbers comparable, nothing left out, still means somebody goes through it by hand. Find the rest. Check what's true. Toss the duplicates. Then do it again next month when it's all changed.

For years the fix was just to do that work, or to reach for a tool that promised to. Every one of them dead-ends the same way.

Write a custom scraper. Works until the site changes. Now you're a scraper maintenance engineer, which is not what you signed up for.

Use a scraping API. Faster to start, but you're still writing extraction logic per site. You traded Playwright for an API call, the core problem, one engineering project per dataset, is unchanged.

Use a pre-built data provider. Great if what you need matches what they index. Companies, people, research papers; covered. GPU retail prices? Restaurant data? Startups funded this morning? You're on your own.

That was tolerable when a person was the bottleneck. It isn't anymore, because now it's the agent's turn and an agent is only as good as what you feed it. Give it a clean, complete, current set of facts and it runs circles around any of us. Give it fifty tabs of half-right pages and it does what we'd do, badly: skims, guesses, forgets what it read three pages back, and hands you a table you can't trust.

That's the actual gap. It isn't that the web is hard to search, agents search fine. It's that nobody's job is to turn what's out there into data you'd bet on: complete, checked, current. So what if you could just ask for it?

What if the input was just a sentence?

Instead of just making a better scraper, the real question is: why is anyone, you or your agent, still putting this together by hand? You already know what you want, and you can say it in a sentence: "YC companies that are hiring engineers, with their funding stage and headcount." That's the whole ask.

That's BigSet.

You describe the dataset; agents go build it from the live web: working out the columns, finding the right entities, checking each one against real sources, verifying them and dropping the duplicates. Set a refresh cadence and it keeps itself up to date. What comes back isn't a pile of links. It's data you can trust: complete, checked, current.

And since the input is just a sentence, something an agent can write as easily as you can, your agents can ask for data the same way you do, instead of rebuilding the same shaky table every time they need one.

How We Built It

Two phases. Schema inference, then population.

Schema Inference

Your sentence goes to Claude Sonnet via OpenRouter. Claude infers the schema: what columns, what types, what's the primary key, and where agents should search.

"Startups that got funded today" becomes six columns. Startup URL as the primary key, startup name, funding amount, funding round, investors, industry. All from one sentence.

The schema is editable, you can add and remove the columns that you need.

Dataset Generation

Solving the context problem here is of paramount importance, the obvious approach of having one agent work on 20 rows works fine. But as you build our larger datasets with 100+ rows.

By entity 30, the agent is forgetting what it found for entity 5. Rows come back with wrong data or missing fields. Not because the model is bad, but because the context window is full of stale information from earlier research. We kept running into this and the quality was unacceptable.

So we built a two-tier, orchestrator - subagent system.

An orchestrator agent takes the schema and searches broadly, it has access to . Breadth-first discovery. When it finds a candidate, it hands the primary key (usually a URL) to a dedicated sub-agent.

Each sub-agent gets one entity, one job, and a tool budget. It confirms the URL actually matches the entity, extracts data for each column, writes a row, and reports a short summary back to the orchestrator. That feedback loop matters: each sub-agent's findings help the orchestrator discover more candidates, so discovery improves as the dataset grows.

Isolated sub-agents prevent context rot. Clear roles prevent drift. Limited budgets prevent spiraling. The result is less hallucinated data and consistent quality across the whole dataset.

Sub-agents run in parallel. Rows stream into your table in real time via Convex with reactive queries. No polling, no refresh button. You watch the dataset build itself.

Security

Your agents are visiting arbitrary web pages, and any of those pages could carry adversarial content designed to hijack the agent through prompt injection. If a malicious page convinces your agent to write rows into a different dataset, that's a real problem.

The standard defense is a system prompt: "only ever write to the authorized dataset." This is security theater. It holds right up until the prompt gets overridden, which is the entire point of prompt injection.

We took a harder path. Every agent tool in BigSet is wrapped in a factory function that captures the dataset ID in a closure. The LLM never sees the ID. Can't reference it. Can't change it.

function createInsertTool(datasetId) {
  return (data) => db.insert(datasetId, data)
  // datasetId lives in the closure. Invisible to the model.
}

Even if a poisoned page fully hijacks the prompt, it is only limited to writing to it's it's own dataset. The capability boundary is in the code, not the prompt. We also enforce update quotas atomically in the database, so straggler agents can't exceed the row limit you set.

How Is This Different From Exa Websets?

Exa built a product for structured B2B data. People, companies, research papers, articles, all backed by a pre-built index. If your use case lives inside that index, it works well, and it'll often be faster than we are because there's nothing to research at query time.

BigSet makes a different trade. It isn't limited to a fixed index, because it doesn't have one. Ask for GPU retail prices, restaurant data, startups funded this morning, or a competitor feature comparison, and BigSet sends agents to the live web to research each entity individually. The cost of that generality is real: per-entity research is slower and more expensive than reading from an index that's already built. The payoff is reach. If the data exists on the web, BigSet can assemble it, including datasets nobody has pre-indexed and datasets that are only true today.

The other practical difference is check-ability. Every row BigSet produces ships with the source URLs it was built from, so you can audit any cell back to where it came from instead of trusting a black box. That's the honest version of "trust": not a promise that every row is verified by agents.

Try It

BigSet is open source and runs locally. Install the CLI and start it:

npm install -g @adamexu/bigset
bigset

Drop in your API keys when the app opens and you're set in a couple of minutes. Then build a dataset from a single sentence:

bigset create "fintech startups in the bay area" --rows 20 --wait --csv fintech.csv

And the part we're most excited about: your agents can run that exact command. Claude Code, Codex, Hermes, OpenClaw, whichever one you use, just point it at BigSet. It builds the dataset, exports the CSV, and reads the rows back to reason over them. No plugin, integrations, nothing to wire up. The agent that needs the data can finally go get it.

Want to run it from source or poke at the internals? Clone the repo and make dev:

git clone https://github.com/tinyfish-io/bigset
cd bigset && make dev

It's experimental. Schema inference isn't always perfect, and some topics work better than others. We're shipping in the open, so things will change.

Get the repo: https://github.com/tinyfish-io/bigset

Go build a dataset that didn't exist yesterday.