Prompt Compression and Caching

In May 2023, Anthropic expanded Claude's context window from 9,000 to 100,000 tokens and celebrated by putting The Great Gatsby inside it.

The larger window meant users could give a model more information to work with, but agentic applications also started adding context themselves. As an agent gathers results from web searches or findings from delegated work, the conversation grows and uses more of the context window.

Prompt caching and prompt compression are two methods to manage that growing context. In this article, I'll compare both on the same agent task to show how each method works.

What is prompt caching?

Prompt caching is an inference technique for reusing computation from earlier requests. When the beginning of a prompt, its prefix, matches a cached entry, those tokens can receive a reduced input rate while the model generates a fresh response.

The model still receives the same information, so caching introduces no quality tradeoff from removing context. Cached tokens still occupy context capacity.

What is prompt compression?

Prompt compression is a context-management technique that reduces the text sent to a language model by removing material or replacing it with a shorter representation. The model receives that compressed version rather than the full original prompt.

For instance, say you're an engineer using the TinyFish APIs as a part of your agent. After the Search API finds product URLs and the Fetch API retrieves cleaned page content, an agent might use this monitoring instruction:

Monitor the A14 laptop, a portable computer for everyday work.

Alert when SKU A14-16-512 is in stock below USD 1,200. We only want

alerts for this exact SKU when it is available below that price.

Include the SKU, price, currency, source URL, and observation timestamp.

A compressor can shorten it to:

Alert when SKU A14-16-512 is in stock below USD 1,200.

Include the SKU, price, currency, source URL, and observation timestamp.

This removes the product description and repeated condition while retaining the alert rule and required output fields.

Compression vs Caching

I wanted to test prompt compression against prompt caching and see which one works better for agentic workflows. Ideally, the agent would complete the same task accurately while spending less. I also wanted to see whether either approach made its responses start sooner.

To make the comparison fair, I kept the task, evidence, model, and tools the same and changed only how the prompt was handled:

• Regular prompting: Send the full prompt with neither compression nor caching.
• Compression only: Shorten the prompt before every request, with caching disabled.
• Caching only: Keep the full prompt and cache its stable prefix.
• Compression and caching: Compress once, then cache the shorter prefix.

I modeled the test on a TinyFish price-monitoring workflow, using GPT-6 Astra with low reasoning effort. Local tools returned three fixed, fictional retailer offers, so changing web data would not affect the comparison.

The task: The agent had to find the cheapest in-stock SKU A14-16-512 below USD 1,200.

Success meant rejecting a cheaper wrong variant and an unavailable offer, then preserving the exact price, currency, source URL, and observation timestamp in both the recorded alert and final answer.

Both compression methods used the same local extractor, which added no model-call charge. Usage confirmed that caching was active only in the intended methods.

The table includes the first cache write in mean cost per successful run. Response timing is median time to first token (TTFT), measured from sending a request to its first generated text or tool-call argument, across all 12 requests per method.

My findings were:

• The combination approach cut cost by 64.7% against regular prompting.
• Caching alone saved 51.8%
• Compression alone saved 33.3%
• All 12 runs returned the correct alert

What does the LLM bill include?

The LLM bill covers input and generated output, including billed reasoning tokens. Check these categories, with caching priced separately where the provider supports it:

• Ordinary input: Tokens charged at the normal input rate.
• Cache writes: Tokens stored for reuse, charged at the write rate.
• Cache reads: Reused tokens charged at the cached-input rate.
• Output: Generated tokens, including billed reasoning.
• Other charges: Tool execution and model-based compression can add separate costs.

Here is the breakdown across three runs per method, using GPT-6 Astra's published model rates. OpenAI's total input includes ordinary input, writes, and reads.

The input and output totals stayed the same, but reading 68.85% of input from cache cut the bill by 51.8%. Token totals alone cannot tell you what a run costs.

How prompt caching works

Prompt caching skips repeated input processing. It does not skip generating the next answer.

Prompt caching reuses prefix computation

During prefill, a transformer calculates attention state for input, stored as key-value (KV) states. Prefix caching skips that work for matching input blocks, identified from their tokens and prior context in the vLLM prefix-caching design.

During decode, the model generates output tokens. That is why TTFT can improve without an equivalent reduction in full task duration, which also includes output generation and tool latency.

Provider-managed prompt caching vs self-managed KV-cache reuse vs semantic caching

Provider-managed caching may use prefix/KV-state reuse underneath its API, while self-managed caching puts that mechanism under your inference service's control.

Semantic caching instead returns a previous answer and must check whether its data is still current.

KV reuse does not guarantee identical generated answers. A semantically cached price alert can be stale if the next Fetch result shows a price or availability change.

How prompt compression works

A prompt compressor selects parts of the input to keep or rewrites it as a summary before sending the result to the main model.

Methods of prompt compression

There's quite a few ways to compress prompts. You need to pick the method according to what context you are willing to lose:

• Token selection: LLMLingua uses a smaller model to remove less useful tokens. Microsoft's LLMLingua project report describes up to 20× compression in its evaluated settings.
• Summarization: Replace earlier conversation with a shorter account of decisions and unresolved work. Count the summarizer's calls and preserve the source needed to check its summary.
• Truncation: Drop older or lower-priority material at a boundary. A sliding window needs a separate place for constraints that remain binding.
• Structured extraction: Retain selected fields or exact sentences. From a cleaned Fetch result, keep the offer fields needed for the alert rather than repeated product descriptions.

When prompt compression hurts quality

Prompt compression isn't always the best choice since it can cause failures when it removes a required detail. The original LLMLingua experiments evaluated specific reasoning and language datasets, so their results do not establish whether a compressed product record preserves the right offer.

For example, if your original request was "Find me a laptop under $1,200 across the Apple M series chips" and the compression eliminates the "M series" part, you'd get irrelevant results listing older Intel models since they match your $1,200 limit.

Does prompt compression break prompt caching?

Yes, if compression keeps changing the beginning of the prompt. The cache can reuse only the matching text before the first change. Everything after that point needs fresh processing.

1. Compressing per query can lead to cache miss

Suppose a compressor keeps only the product details relevant to the current SKU. Each product then creates a different reference before the cache boundary. The changed text needs fresh processing, although matching text before the first change can still be reused.

So, I'd check the bill before shipping such a change. The cache-aware compression paper shows why: on τ-bench retail, the shorter per-query prompts cost more than leaving the original cached prompt alone.

The chart indexes the original cached prompt's cost to 100. New writes and lost discounted reads outweighed the token savings in this test. The paper's cache-aware static approach cost 7.9% less and matched the original agent's observed task reward.

If the reference must change for every query, measure the lost cache reuse before treating fewer tokens as a saving.

2. Compress static content once, then cache the result

I would compress a stable product reference once, then reuse it until the source changes. Keep changing observations after the boundary:

product-matching policy and tool definitions

compressed product reference

CACHE BOUNDARY

current product query

fresh Search URLs and Fetch observations

If the full prefix up to the boundary falls below the provider's minimum length, it cannot be cached.

In a separate LongBench-v2 question-answering experiment across 16 test settings, the paper's static approach averaged 49% savings against cache-only and 90% against vanilla. Those savings came with lower average quality scores: 0.66 for compression plus caching, compared with 0.75 for caching alone and 0.73 without either technique.

What does prompt caching cost on Anthropic, OpenAI, and Gemini?

These standard text API rates apply to the named models, with GPT-6 Astra limited to requests up to 272,000 input tokens. Gemini rates shown are introductory through December 31, 2026. Retention is also called time to live (TTL).

Anthropic charges for the write and discounts every read by 90 percent

On the Claude Messages API, put cache_control at the end of the content you expect to reuse. This abbreviated request shows its placement. Replace the sample reference with enough real material to meet the model's minimum:

{

"model": "claude-sonnet-5",

"max_tokens": 300,

"thinking": {"type": "disabled"},

"system": [{

"type": "text",

"text": "Stable instructions and versioned reference content go here.",

"cache_control": {"type": "ephemeral", "ttl": "5m"}


}],

"messages": [{"role": "user", "content": "The changing question goes here."}]

}

Sonnet 5 charges 1.25× ordinary input for a five-minute write or 2× for an hour, with reads at 0.1×. The Claude caching reference lists supported breakpoint placement and minimum lengths.

Sonnet 5 is also available through Claude in Bedrock. Check that integration's request format and Bedrock cache support before adapting the example.

OpenAI caches automatically, and GPT-6 Astra supports explicit breakpoints

OpenAI enables caching automatically on supported models. On GPT-6 Astra, explicit mode can mark the end of stable content:

{

"model": "gpt-6-astra",

"prompt_cache_options": {"mode": "explicit", "ttl": "30m"},

"input": [{

"role": "developer",

"content": [{

  "type": "input_text",

  "text": "Stable reference content exceeding the model minimum.",

  "prompt_cache_breakpoint": {"mode": "explicit"}

}]


}, {

"role": "user",

"content": "The changing question."


}]

}

An explicit-only request without breakpoints disables reads and writes on these models. prompt_cache_key separates accounting on GPT-5.6 and later. Earlier models use stable keys to help route related requests to the same cache and have different retention controls, as documented in the model-specific caching rules.

Gemini discounts cached input 10x and meters explicit caches by the hour

Gemini 3.8 Flash enables implicit prompt caching by default.

Explicit Gemini caching is a beta feature that uses a named cache resource with a TTL through the Generate Content API. The Interactions API supports implicit caching, not those explicit objects. Include hourly storage in the bill, and delete unused caches or shorten their TTL.

When does a cache write pay for itself?

For Sonnet 5, a five-minute cache write pays back after one full reuse. A one-hour write needs two, before the entry expires.

Let's do some math here to understand the price economics better.

For a fixed prefix, let n be total uses including the first write. Express costs as multiples of processing that prefix once without caching. Under Sonnet 5's published rates, that'd mean:

no cache: n

five-minute: 1.25 + 0.1 × (n - 1)

one-hour: 2.00 + 0.1 × (n - 1)

With one reuse, the five-minute option costs 1.35 times the ordinary prefix cost, compared with 2 without caching. The one-hour option costs 2.10 at that point. After two reuses, its 2.20 cost falls below the uncached total of 3. An expired entry that must be rewritten changes the calculation.

In Don't Break the Cache, controlled DeepResearch Bench experiments reported cost reductions of 41–80% and TTFT reductions of 13–31% across the evaluated configurations. The authors also found that naive full-context caching could increase latency.

Why does prompt cache not hit at times?

A cache miss occurs when the provider cannot find an available entry matching an eligible prefix.

Keep the cached part of the prompt unchanged

Matching depends on the provider's tokenized, model-visible input, not equivalent meaning or necessarily the raw HTTP bytes. A JSON key-order change matters if it changes the rendered prompt or serialized tool content, rather than merely the HTTP representation.

Diagnose a miss against the actual request:

• Changing prefix: move a timestamp or request ID behind the reusable boundary. Preserve the content before it.
• Tool changes: compare tool definitions and their ordering between calls. A tool edit can change an early prefix.
• Short prefix: compare the marked prefix with the selected model's minimum. A marker on a short prompt does not make it cacheable.
• Concurrent cold requests: let an initial response begin before testing reuse on Claude. Simultaneous first requests may arrive before an entry is available.
• Boundary discovery: Claude checks up to 20 content-block positions, including the cache breakpoint itself. Place another supported breakpoint when the reusable boundary falls outside it.
• Expired retention: compare the request interval with TTL. Keep the write and read usage associated with the same prompt version.

On the Claude API, a marked prefix below the minimum is processed without caching and can return no error. Repeated writes with few reads need attention too: compare request spacing and prefix changes before paying for another write.

The Claude cache diagnostics document these provider-specific limits.

Check the usage fields on every response

Calculate savings from the provider's reported token categories, since ordinary input and cache reads carry different prices. This illustrative Claude usage object is not benchmark output:

{

"usage": {

"input_tokens": 120,

"cache_creation_input_tokens": 0,

"cache_read_input_tokens": 4096,

"output_tokens": 80


}

}

In Claude's usage accounting, input_tokens excludes reads and writes, so total input here is 120 + 0 + 4096 = 4216.

The OpenAI usage calculation subtracts cached and written subsets from input_tokens to price ordinary input. Using Claude's accounting formula here would double-count them.

Audit your token bill

Use this prompt with a seven-day usage export to find prefixes worth caching and writes that earn no later reads. Include request timestamps and applicable model prices, keeping raw customer prompts out of the report.

Audit the supplied seven-day model-usage export. Do not change production.

Normalize input accounting by provider. For Claude, total input equals

ordinary input plus cache writes plus cache reads. For OpenAI, subtract

cached and cache-write subsets from total input to obtain ordinary input.

Preserve model and pricing-tier distinctions.

Calculate:

- Input-to-output cost ratio, including cache-write premiums.

- Cache-read share of total input, by model and prompt version.

- Cache-read tokens divided by cache-write tokens within each TTL window.

- Share of cache writes with no observed later reads.

- Share of attempted cached requests below the model's minimum prefix.

- Cost per successful task, including failed attempts and compressor calls.

Use local hashes or approved metadata to compare prefixes. Flag timestamps

or request IDs appearing before reusable instructions. Check whether tool

ordering or a summarizer rewrites the reusable content between turns.

Account for retention expiry and concurrent first requests.

Do not infer write-without-read events from aggregate token counters alone.

If prefix IDs or timing are missing, mark that finding unavailable. Do not

infer success from a completed HTTP request. Explain missing data and which

conclusions it prevents.

Report percentages and ratios only. Omit platform identity, customer content,

absolute spending, traffic counts, and private file paths. Keep input-billing

improvement separate from measured task success and latency. Recommend a

controlled test for each proposed change, with a pass condition and rollback.

FAQ

What is prompt caching?

Prompt caching is reuse of saved input computation for a matching prompt prefix, usually at a discounted input rate.

How does prompt caching work?

The server saves state for an eligible prefix and reuses it when a later request matches before expiry.

What is prompt caching in Claude?

It is Claude's reuse of saved prompt-prefix computation, configured through cache_control or its supported automatic caching control.

Does prompt caching reduce latency?

A cache hit can reduce TTFT by avoiding repeated prefill. Our benchmark did not show that improvement, and tool execution and output generation still take time.

How do you implement prompt caching?

Place stable content before changing input, configure the model's cache controls, and verify reuse in response usage before comparing costs.

Does Claude Code use prompt caching?

Yes. Claude Code's prompt caching documentation explains its reuse of instructions and conversation context. On supported versions, /usage reports cache statistics and /context reports context occupancy.