The Prefix Is Everything

An agent sends its whole prompt again on every step. A prompt cache makes that repeat cheap, but only up to the first byte that changed. Everything after that byte is billed and computed again, as if the model had never seen it.

While working on cache improvements for Neo, our AI security agent, I started logging every cache miss and what caused it. Caching was on the whole time. The misses came from habits that looked harmless. Here is the model I now carry in my head, a toy you can break, and five rules that came out of it.

What an agent sends

A chat app sends one message and gets one reply. An agent runs a loop. The model asks for a tool, the tool runs, the result goes back, and the model asks again.

Each trip is a new request, and the model remembers nothing between them. So every request carries everything, in this order:

1. Tools: the name, description and input schema of every tool the model may call.
2. System prompt: who the agent is and the rules it follows.
3. History: the task, then every tool call and tool result so far.

Step 30 re-sends steps 1 to 29.

Take a toy agent with 11,000 tokens of tools and system prompt, adding 2,500 tokens of history per step. By step 30 a single request is 86,000 tokens. Across the 30 steps the model reads 1.49 million input tokens. At an illustrative $3 per million, that is $4.48 for one run.

With a cache that hits every time, only the new tokens are fresh. The run costs about $0.74, or 83% less.¹

How the cache matches

Prefix caching: the provider keeps the model’s work for prompts it saw recently, and reuses it when a new prompt starts the same way.

The match runs from the first token forward and stops at the first difference. It has to. The model’s work for token 500 depends on tokens 1 to 499.² Change token 3 and the work for every token after it is wrong, so it is thrown away.

The cache can reuse a prompt only up to the first changed byte.

A cached token is cheap. On Anthropic’s API a cache read costs a tenth of the normal input price, and a write costs 1.25 times.³ OpenAI discounts cached input by up to 95%.⁴ DeepSeek charges 30 to 50 times less for a hit than a miss.⁵ The hit is also faster, because the model skips work it already did.

Break it yourself

Each button sends the agent’s next request after one change. Grey blocks come from the cache. Red blocks were cached before and are now billed again.

Try this order:

1. Click Next step a few times. The grey grows and only the new turn is fresh.
2. Click Put the time in the system prompt. The tools survive; everything after them turns red.
3. Click Drop the oldest turn three times. Every click misses.
4. Click Trim history to 60%, then Next step twice. One miss, then hits again.

What decides a hit

Seven things decide whether the next request matches the last one.

The last two differ by provider. As of October 2026:

OpenAI and DeepSeek cache by default. Anthropic needs a cache_control field: one at the top level caches automatically, or you place up to four breakpoints yourself. A one-hour lifetime is available there at twice the normal write price. OpenAI offers longer retention as an option. vLLM’s flag is enable_prefix_caching, and it evicts the least recently used blocks first.

Automatic caches are best effort. OpenAI’s docs say outright that even a cache key does not guarantee a hit.

Five rules

1. The prefix is everything. One changed byte early re-bills everything after it. Know what sits highest in your prompt and how often it changes. On open-model servers, also read the chat template. Some, like Qwen’s, render the tool list inside the system message, right after your system prompt.⁶ There, a timestamp in the system prompt re-bills the tools as well.

2. Freeze the tool list. Pick the tools when the session starts and do not touch them until it ends. If some tools are needed only rarely, reach them through one stable dispatcher tool, such as run_extra(name, args), instead of adding them mid-run.

The open-source agent oh-my-pi works this way. Its tool list keeps the same shape all session, and tools check at run time whether they may run.⁷ The cost: the model sees less detail for the rare tools, and you validate their arguments yourself.

3. Append, never edit. New instructions and new state go at the end, as a message or a tool result. “The time is 14:02” belongs in the latest message, not the system prompt.

The cost is a longer history, with old state still sitting above the new. That is cheaper than a miss. Providers are moving this way too: Anthropic now accepts system messages in the middle of a conversation on some models, without breaking the cache above them.³

4. Trim in batches, not every step. Context fills up and something has to go. Drop the oldest message every step and the cut moves every step, so every step misses from that point down.

Instead, let history grow to a high mark, cut it to 60–70% in one go, then hold. One miss per cut instead of one per step.

In the toy agent, capped at 80,000 tokens over 100 steps, trimming every step re-bills 4.8 million tokens and costs $15.61. Trimming in batches re-bills about 270,000 tokens and costs $3.20. The cost: between cuts you carry more context than you strictly need. oh-my-pi’s pruning is cache-aware in the same spirit. It blanks an old tool result only when little cached context sits after it, or when the cache has expired anyway.⁷

5. Measure breaks, don’t guess. Every provider reports how many input tokens came from the cache. That number tells you that you missed. It does not tell you why.

So fingerprint each request by section, and compare it with the last one:

import { createHash } from "node:crypto";

type Sections = { tools: string; system: string; messages: string[] };

const hash = (bytes: string) =>
  createHash("sha256").update(bytes).digest("hex").slice(0, 12);

const fingerprint = (s: Sections) => [
  hash(s.tools),
  hash(s.system),
  ...s.messages.map(hash),
];

function firstChange(prev: string[], next: string[]): string {
  const i = prev.findIndex((h, k) => h !== next[k]);
  if (i === -1) return "none";
  if (i === 0) return "tools";
  if (i === 1) return "system";
  return `message ${i - 2}`;
}

Hash the bytes you actually send, not your objects. Two equal objects can serialize differently, and that difference is exactly what you are hunting.

On every miss, log which section changed first and how many tokens it cost. Causes you would never guess show up in the first day of logs. A map that serializes in a different order. A field that is “constant” except when it isn’t.

Treat the prompt like a log file: write at the end, and never above.