Context Rot: How Long Inputs Degrade LLM Accuracy in 2026

Key Takeaways

• Context rot is the measurable decline in LLM output quality as input grows. It starts long before the context window is full.
• Chroma's July 2025 report tested 18 frontier models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. Every one degraded on longer inputs, even on trivial copy-and-retrieve tasks.
• Length is the symptom, retrieval quality is the variable. The unit of rot is the retrieval, not the turn: one fetch returning 3% article content does more damage on turn one than 40 clean turns.
• Cleaning input at the retrieval layer is the highest-leverage fix. Built with TinyFish Search and Fetch, Mohammad Varmazyar's open-source web-research tool cut one three-page benchmark from 23,851 estimated tokens to 1,608, a 93.3% reduction.

Nothing crashed. Your agent ran for 20 minutes, read nine pages, and quietly misreported a price on turn 5 that it had read correctly on turn three.

That's context rot, the steady loss of LLM accuracy as input grows, arriving long before the token limit does.

Chroma, which builds an open-source retrieval database, found it in all 18 models it tested. Most coverage stops at that headline and blames session length, but Chroma's four controlled needle experiments say otherwise: every one varied what sat inside the input, not just how much.

Below we cover the mechanism behind context rot, the symptoms to watch for in production, what noisy input costs you, and five techniques that prevent it.

What Is Context Rot?

Context rot is the measurable decline in an LLM's output quality as its input grows longer, even when the task itself stays exactly as hard. More tokens in, worse answers out.

Chroma researchers Kelly Hong, Anton Troynikov and Jeff Huber formalized the term in a technical report published in July 2025. Their design is what makes it credible: they held task complexity constant and varied only input length. Most long-context benchmarks fail here, because longer inputs usually mean harder questions too. Their conclusion was blunt. Models don't use their context uniformly, and performance grows increasingly unreliable as input length grows.

Here's the distinction that trips teams up. Context rot isn't context window overflow. Overflow is a hard failure at the token limit. Rot is a gradual slide that begins far below it. A model with a 200,000-token window doesn't hold full accuracy until token 199,999 and then fall off a cliff.

That's why "we upgraded to a bigger window" solves nothing. Capacity was never the constraint. What your agent can actually use is a much smaller number than what it can technically accept, which is the argument we made in give agents better context and again in our work on context versus tokens in agentic coding.

One more correction, because most coverage of Chroma's report misses it. Rot is usually described as a session-length problem: run long enough, accumulate enough turns, quality decays. Length is real, but it's a proxy. Two of Chroma's four experiments varied the content of the input rather than its size, and both moved the results. Rot is what happens when the signal-to-noise ratio of your input falls. Length is just the most common way that ratio falls, which puts the lever in your retrieval layer, not your session policy.

What Causes Context Rot

Three mechanical things happen to a model as its input degrades. None of them are mysterious, and all three sit downstream of a retrieval decision you made.

Cause #1: The Model Does Two Jobs at Once

Give a model only the relevant text and it does one job: reason. Bury that text in 100,000 tokens of other material and it does two: find the relevant part, then reason over it. Accuracy drops because you added a task, not because you added tokens.

Chroma measured this with LongMemEval. Models answered questions from a chat history in two conditions: a focused prompt averaging around 300 tokens, and a full prompt averaging around 113,000. Every model family scored significantly higher on the focused version. Same question, same answer present in both, very different results.

Cause #2: Position Changes What the Model Notices

Where information sits affects whether the model uses it. The classic finding is Lost in the Middle (Liu et al., 2023): accuracy dipped when the relevant passage sat mid-input rather than at either end.

Chroma adds a nuance most write-ups skip. In their needle-in-a-haystack task, 11 needle positions produced no notable variation. In their repeated-words task, accuracy was highest when the unique word sat near the beginning. Position sensitivity is real but task-dependent, so test your own workload instead of assuming the middle is always the danger zone.

Cause #3: Bad Retrieval Poisons the Window

This cause gets the least attention and matters most, because it's the one your code controls directly.

Noise isn't neutral padding. Chroma wrote four distractors for one question: sentences that were topically close but didn't answer it. A single distractor lowered performance against the baseline, and four compounded it. Read that experiment for what it is. A distractor is precisely what a mediocre retriever returns, topically similar and semantically plausible but not an answer. Chroma didn't only measure length. It measured retrieval quality and found that worse retrieval accelerates rot.

The similarity experiment says the same thing from another angle. Chroma varied how closely each needle matched its question by embedding cosine similarity, and lower-similarity pairs degraded faster as input grew. Cosine similarity between query and passage is the exact quantity your vector search optimizes. Your retriever's precision isn't a separate concern from context rot, it's an input to it.

Failure modes then split by model family. Claude models had the lowest hallucination rates and tended to abstain under uncertainty, saying no answer could be found. GPT models hallucinated most, producing confident wrong answers when distractors were present. If you've read that Claude confidently invents facts at long context, that's the opposite of what the data shows.

How Context Rot Shows Up in Production Agents

You won't see an error. You'll see quality drift.

Facts from early context come back wrong: the agent quotes a price it read on turn two, and by turn 15 it has drifted. Constraints set in the middle of a long instruction block get ignored. Answers are sharp on the first three tool calls and vague by the tenth. And the tell that catches most teams: the same task passes on a short input and fails on a long one.

Agents are especially exposed because they generate their own noise. Every search returns snippets you didn't need, every page fetch drags in navigation and ad slots, every retry appends another failed attempt. Nobody wrote a 90,000-token prompt. The agent assembled one, one tool call at a time.

That's the structural difference between an agentic workflow and ordinary automation. A script's input is fixed. An agent's input is whatever it picked up along the way, and if you can't see what it picked up you can't debug the drift. Hence observability in your agent stack.

But notice what "along the way" means. Turn count isn't what hurts you. What each turn brings back is. An agent running 40 turns on clean, scoped retrievals can finish in better shape than one that runs three turns and ingests a single 165,000-character page at 3% signal. Session length is easy to measure, so teams optimize it and get modest results.

Pro tip: Before you shorten your sessions, audit one retrieval. Print the character count of a single fetch next to the length of the content you wanted. If the ratio is 10 to 1 or worse, your rot problem is a fetch problem and session trimming won't fix it.

The Price of Noisy Context

You're not paying for answers. You're paying for navigation bars, cookie banners and trending-story rails, billed at the same rate as the paragraph you actually wanted.

We ran the numbers in 80% of your web fetch returns junk in May 2026, pulling 15 articles from five publishers inside the same five-minute window. On one Daily Mail article with a body of roughly 4,300 characters, TinyFish Fetch returned 4,673 characters at around 92% article content. One competing service returned 164,986 characters at around 3%, padded with site navigation, a weather widget, 60-plus trending links and ad slots.

At roughly four characters per token, that's about 1,170 input tokens versus about 41,000 for the same article. Thirty-five times the cost, for the same information.

The bill is the smaller half of the problem. Those 40,000 junk tokens aren't inert filler. Trending headlines and related-article rails sitting beside a news story are textbook distractors: same topic, same vocabulary, no answer to your question. That's the exact condition Chroma showed degrades accuracy, delivered in one API call. Our fetch quality evaluation put it plainly: a dirty fetch hurts you twice, once on tokens and once on reasoning.

This is where the session-length framing breaks down. That page is turn one. Nothing has accumulated, there's no history to summarize away, and the window is already 97% noise because of a decision made in the retrieval layer. Every later turn reasons on top of it.

That's the job TinyFish Fetch is built for. Point it at a URL and it renders the page in a real browser, strips site chrome before extraction, and returns clean Markdown, JSON or HTML. Pages that aren't really content, like bot challenges and empty responses, come back as errors rather than as source material. Nothing extra enters the context window, the bill, or the model's memory. More in production-grade web fetching.

Clean inputs, one call. Search and Fetch are free on every plan, at any wallet balance including $0. Grab an API key and run one URL through both.

How to Prevent Context Rot

Five techniques, ordered by leverage. The first two control what enters the window. The last three manage what's already in there, which helps but is strictly downstream.

1. Cut Your Top-K and Re-Rank

Retrieve fewer but better chunks. Most RAG pipelines default to top-10 or top-20 because recall feels safe, and every marginal chunk past the answer is a distractor by definition. Chroma priced that habit: one distractor lowered performance, four compounded it. Cut your k value, raise your relevance threshold, and re-rank before the model sees the shortlist. Expect fewer confidently wrong answers, and a bigger gain than anything you get from trimming session history.

2. Fetch Clean Pages, Not Raw HTML

Stop putting raw pages into context. With Fetch you can scope extraction using include_selectors and exclude_selectors, passing up to 20 CSS selectors to keep only the region you care about and drop comment threads or signup rails before extraction. See extracting structured data from a website for patterns.

3. Compact Between Phases

Summarize and discard as you go instead of carrying the full transcript forward. After each phase, replace raw tool outputs with a short structured summary and drop the originals. We saw the same effect when we shipped a CLI alongside our MCP server: writing output to disk rather than context left the agent far more room to think. Expect long runs to hold quality further into the session.

4. Move Durable Facts Into a Schema

Keep durable facts in a schema, not in prose scattered through a transcript. A small JSON object holding the entities, constraints and decisions your agent needs costs a fraction of the conversation that produced it, and the model doesn't re-derive it every turn. Fetch returns a structured document tree when you set format: "json", so you can store the fields you need and drop the rest.

5. Budget Tokens by Stage

Give every stage of your agent a token allowance and enforce it. Decide up front what retrieval, page content and history each get, then truncate or compact anything over budget. Batching helps: Fetch accepts up to 10 URLs per request, so you can pull a working set, filter it, and pass only what survives. More in fetching data from multiple URLs and our guide to choosing the right TinyFish API.

Did you know? Fetch serves a cached entry by default when one exists. Pass ttl: 0 when you need a live read, or a positive integer in seconds to set your own freshness window. Freshness is a parameter, not a guess.

How to Detect and Measure Context Rot

Run the same fixed task at increasing input lengths and watch the score fall. That's the whole method, and it's the one Chroma used.

Build an eval set of 20 to 50 questions with known answers. Run each at several padding levels, say 1,000, 10,000, 50,000 and 120,000 tokens, keeping the question and answer text identical and varying only the filler. Any drop across that curve is context rot, because nothing else changed. Then hold length fixed and move the answer between start, middle and end. Accuracy that shifts with placement is the clearest single signal you'll get.

Then measure the input side, the test almost nobody runs. For a sample of 50 real retrievals, log total characters returned against characters you actually needed. That median ratio is your signal score, and it predicts output quality better than average session length does. Track it like a latency metric. It moves when a site changes layout, and you want to know before your accuracy does.

For a reference point, our Fetch quality eval, last updated July 2026, scores pages returned as usable context: TinyFish at 93%, Tavily 80%, Exa 73%, Firecrawl 62%, Parallel 58%. Being honest about the full picture, on that same page Firecrawl completed 96 of 100 BrowseComp tasks without a timeout or error against our 95. Different tools win different rows. For task completion rather than reading, see our Mind2Web results and the third-party accuracy evaluation.

Keep Your Context Clean With TinyFish

Quality tracks the signal-to-noise ratio of what you feed the model, not the size of the window you feed it into and not how long the session has been running. Every technique above is the same move: put less junk in front of the model so more of its attention lands on what you actually asked about. The highest-leverage place to do that is the retrieval layer, because that's where the junk gets in.

The web is where most of that junk originates. Fetch returns clean page content and nothing else, Search finds the right sources before you read them, and both are free on every plan at any wallet balance. Cleaning your agent's inputs takes one call.

Start with one URL. Create an API key, run a page through Fetch, and compare the character count against whatever you use today. If you're building multi-step runs on top of it, read the platform overview next.

FAQs

1. At How Many Tokens Does Context Rot Start?

It depends on the task, and there's no universal threshold. Chroma found degradation across the full range they tested rather than at a single breakpoint. The better question isn't how many tokens but how clean they are: a 5,000-token window at 90% signal will outperform the same window at 10% signal every time. Measure your own curve, and measure your signal ratio alongside it.

2. Does a Bigger Context Window Fix Context Rot?

No. Window size sets the ceiling on what a model will accept, not the point at which it stays reliable. Chroma tested models with windows in the hundreds of thousands to millions of tokens and found all 18 degraded well before those limits.

3. Does Context Rot Affect All LLMs?

Yes. All 18 models in Chroma's evaluation degraded as input length grew, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. Only the failure mode differs: Claude models tended to abstain under uncertainty, GPT models answered confidently and wrongly.