How I built a comprehensive AI trend monitor with real-time web data

A build log about measuring the AI hype cycle honestly with BigSet, the metric that almost fooled me, and the moment the data confirmed my own thesis by accident.

The AI Buzzword Tape, measured and sourced. “We are here” marks today; everything to the right is projected.

TL;DR

I built The AI Buzzword Tape, a comprehensive AI trend monitor with real-time web data using TinyFish’s BigSet, an open-source dataset builder powered by TinyFish’s search and fetch layer.

The AI wave is moving at an exponential pace. Models leapfrog within months, and new harness layers drop sooner than one can account for. As an AI investor, the core challenge of my job is keeping up with the frontier, and X and Substack posts can only go so far. The goal was simple: turn a vibe-based AI hype-cycle chart into something measured, sourced, and refreshable.

BigSet handled the open-ended research layer by discovering terms, adding definitions, assigning categories, and attaching source links. A deterministic Hacker News counter handled the time-series math. BigSet is well suited to curation, discovery, and sourcing, while precise time-series measurement is better handled in deterministic code.

The result was a sourced AI buzzword tape across 14 terms, measured every quarter from 2023 to 2026, with every figure traceable back to a Hacker News source URL.

The context

Never in any of the technology cycles have we seen buzzwords come and go at such velocity. However, there is a lot of value in understanding what is trending and why.The AI buzzword cycle is one of the most useful signals in venture.

Hot terms mark the current bottleneck. And they are late to the breakthrough, early to the money: the tech works before it has a name, so by the time a word is everywhere, the capital is already moving in.

Read the tape left to right and every term walks away from the model:

language → orchestration → economics → physics

That was the argument I wanted to put on a wall.

But most charts like this are vibes. They may look right, but there is nothing underneath that a reader can check. If someone asks why “context engineering” peaks for one quarter while prompt engineering for 3, “it is important” is not a good answer.

So I set one rule before drawing a single curve:

Every wave needed to be measured, sourced, and defensible.

An opinion is something you argue. An instrument is something you can point at and refer to for decisions.

What I built

I built a measured AI buzzword tape with three jobs:

1. Curation: Which AI terms belong on the tape? What category are they in? What do they mean? What source supports them?
2. Measurement: When did builder interest actually peak? How large was the wave? Did it decay?
3. Rendering: How do those measured stats become a chart people can understand at a glance?

BigSet handled the first layer by building the initial research table and attaching definitions, categories, emergence years, and source URLs to each term.

The measurement layer was separate. I used a deterministic Hacker News counter to query exact phrases across quarters and sum story upvotes.

The renderer then fused the two layers and drew each wave from three measured numbers:

• peak position
• height
• width

The smooth curve is modeled from measured stats.

Where TinyFish fit

BigSet is TinyFish’s open-source dataset builder.

It runs on TinyFish’s web layer: its research agents reach the live internet through TinyFish Search and Fetch. That is why the workflow only needs a TinyFish key and an OpenRouter key to run locally.

No scraper to write. Nothing to host.

For this project, TinyFish powered the live research layer behind BigSet. BigSet used that layer to discover terms, verify facts, collect source links, and return structured rows.

That part worked extremely well.

In a few minutes, I had a table of AI terms with definitions, categories, emergence years, and traceable URLs. That is the open-ended curation problem: what exists, what is it, and where is the proof?

Then I looked at the peak numbers, and they were wrong.

Not randomly wrong. Informatively wrong.

BigSet gave several terms the same peak quarter and suggested that almost everything “had not declined yet.” When I traced the issue, the reason made sense: each research agent runs on a tight tool budget, around six web actions per row. Building a real popularity-over-time curve requires querying many time windows and normalizing them. That does not fit inside a short per-row budget.

So the agent approximated.

That budget is a feature. It keeps BigSet fast and cheap across many rows. But it also clarified the right architecture:

Use BigSet for open-ended discovery, curation, and sourcing. Use deterministic code for precise time-series math.

That split made the project better.

BigSet kept the job it was great at. The counter handled the math.

What worked

1. The obvious metric was wrong

My first plan was simple: count how often each term appeared on Hacker News per quarter.

That seemed reasonable. If a buzzword peaked and faded, mentions should rise and fall.

Then I checked “prompt engineering.”

The result did not match the story I expected.

Mentions did not collapse. They stayed high. In some quarters, they were higher than the supposed peak.

That was the first real insight:

Mentions measure presence. Presence does not fade for terms that win.

“Prompt engineering” did not disappear. It became vocabulary. It moved into job descriptions, product pages, and everyday discussion. Once a term becomes normal, mentions can stay high even when excitement has faded.

I needed a metric that measured excitement, not presence.

Hacker News already had one: upvotes.

When a term feels novel, people upvote it. When it becomes routine, people may still post about it, but they stop treating it as exciting.

So I switched from story count to total Hacker News upvotes per quarter.

Hacker News is not the entire builder market. It is a narrow, opinionated signal. But that is also what made it useful here: marketing language has a harder time surviving unless builders actually find the topic interesting.

That changed the tape.

“Prompt engineering” still appeared often, but its excitement had clearly decayed.

That was the difference between counting vocabulary and measuring attention.

2. The tape made each term more legible

Once the metric changed, the patterns became easier to defend.

“AI agents” became the loudest term on the builder tape, peaking at 5,227 upvotes in late 2025.

“Context engineering” showed a sharp, narrow spike, matching the idea that some terms run hot for only a short period before the field moves on.

“Prompt engineering” became the early fossil: a real peak, then a decay in excitement as the term became part of the background language.

The chart started to feel less like a vibe and more like an instrument.

3. The data challenged the capital narrative

The most interesting result came from the term “tokenomics.”

In capital and social discourse, tokenomics felt like a current darling. But on the builder tape, it was much quieter than expected, peaking around 247 upvotes.

Meanwhile, “AI agents” reached 5,227 upvotes.

That gap became the thesis in data form.

The terms investors talk about and the terms builders upvote are not always moving together. Sometimes capital names a wave before builders care. Sometimes builders make noise before capital has a clean label for it.

That lag is the signal I wanted to study.

Before publishing this claim broadly, I would still want to verify the underlying Hacker News results for “tokenomics,” since the term can also refer to crypto token economics. But the gap itself is the important pattern: the tape can show where capital discourse and builder attention diverge.

Measurable outcome

What I’d do differently

The biggest next step is to let BigSet own the term layer end to end.

In this first version, I still hand-seeded the terms and categories, then used BigSet to verify and structure them. The real unlock would be a weekly BigSet refresh that surfaces new terms as they cross a threshold.

That would turn the tape from a retrospective chart into a living market-intelligence system.

A few improvements I would make next:

• Add a weekly refresh and dated snapshot series.
• Watch terms climb across frames instead of only looking at one final chart.
• Normalize against overall Hacker News activity so raw upvotes are not confused with platform growth.
• Tighten noisy terms like “RL” and “self-learning” with better phrasing.
• Cross-check builder attention against other sources such as GitHub, arXiv, npm, or PyPI.
• Add a second “capital tape” from VC threads, funding announcements, and market narratives, then measure the gap between builder attention and investor attention.

The bigger ambition is prediction.

The current tape measures waves that already crested. The more interesting version would flag a term before it fully crystallizes, while it is still unnamed noise in builder discourse.

That would require features like:

• acceleration of attention
• builder-to-capital lead-lag gap
• phrase crystallization
• bottleneck concentration
• upvote-vs-mention divergence

That is the instrument I actually want.

Recommendation for other builders

The reusable lesson is not about buzzwords. It is about matching the tool to the shape of the job.

Use BigSet for open-ended curation and discovery:

• finding entities
• verifying facts
• attaching sources
• building structured datasets
• refreshing live research layers

Use deterministic code for exact computation:

• time series
• aggregates
• precise counts
• scoring logic
• deduplication

Do not make a short-budget research agent do math that belongs in code.

But also do not write a scraper or manual research loop when an agent swarm can discover, verify, and source the rows for you.

That split is what made the tape defensible.

I did not want a chart I could only defend with conviction.

I wanted one I could defend with a URL.

Try it / Links

BigSet open-source: github.com/tinyfish-io/bigset

Measurement layer:

Hacker News Algolia API, exact-phrase queries, upvotes summed per quarter.

📌 Sign up free → agent.tinyfish.ai

Docs → docs.tinyfish.ai

Open source Cookbook → github.com/tinyfish-io/tinyfish-cookbook