The Cheapest Tokens Are the Ones You Already Paid For
Here is a question worth asking the next time an API bill arrives: how much of what you sent the model did the model actually need to read fresh? In most production applications, the answer is surprisingly little. A system prompt describing the product, a batch of tool definitions, a few dozen few-shot examples, a style guide, a safety policy — all of it is identical from one request to the next. Yet many teams pay full price for that boilerplate on every single call, thousands of times a day.
Prompt caching exists precisely to fix that waste, and in 2026 the economics have become too large to ignore. Anthropic's prompt caching can cut costs by up to 90 percent and latency by up to 85 percent on cached input, and the September release of Claude Fable 5.1 shipped with cache-read prices cut by a further 75 percent. OpenAI applies automatic prefix caching with discounts that commonly land between 50 and 90 percent depending on the model, and Google's Gemini platform offers explicit caching for long-lived contexts. The gap between a well-structured prompt and a carelessly structured one is no longer measured in pennies; for agentic workloads it can be the difference between a profitable product and a money pit.
Why Ordering Is an Economic Decision
Caching works on the simplest possible principle: providers remember a prefix of your prompt and charge a steep discount when you send the same prefix again. "Prefix" is the operative word. Whatever appears first in your request is what gets cached; whatever appears after a cache breakpoint is not. That single fact turns prompt ordering from a stylistic preference into a financial strategy.
Most teams still write prompts the way they write documents, with the most recent and most specific instructions at the top and the stable scaffolding at the bottom. That is exactly backwards for cost. The structure that saves money looks like this:
1. System instructions <- stable, cached 2. Tool / function schemas <- stable, cached 3. Few-shot examples <- stable, cached 4. Static knowledge base <- stable, cached 5. [cache breakpoint] 6. User message <- variable, fresh 7. Retrieved context <- variable, fresh
The rule of thumb is brutally simple: stable content first, variable content last. Everything that will not change between calls belongs above the cache breakpoint; everything that changes per request belongs below it. When you honor that ordering, a long agentic session can send hundreds of thousands of cached tokens at a fraction of their normal price on every single turn.
Cache-First Design in Practice
Translating the principle into a workflow takes more than moving a few paragraphs around. The most important constraint is that cached prefixes must be byte-identical across calls. A timestamp rendered into the system prompt, a randomly ordered tool list, or a serialized user object that includes a changing field will quietly invalidate the cache and erase the discount, even though the prompt looks "the same" to a human.
- Freeze the tool schemas. Sort tool definitions by name or keep them in a fixed order in code, never in insertion order from a dictionary.
- Pin static context above the breakpoint. Style guides, policy documents, and onboarding material belong in the cached region, not injected per request.
- Keep volatile data out of the prefix. Dates, IDs, and per-user fields must live below the cache point, or the prefix changes every call and caching never fires.
- Use explicit cache markers where supported. Anthropic's cache_control breakpoints, OpenAI's automatic prefix detection, and Gemini's explicit cache all reward deliberate placement; read the docs for your provider and mark the boundary explicitly.
- Version prompts like code. A prompt you can diff is a prompt you can debug. When you change the cached region, expect a cache cold-start and plan around it.
A prompt you can version, test, cache, and observe will beat a "better-worded" prompt that gets rewritten ad hoc every sprint. Structure compounds; wording does not.
Where the Big Savings Live: Agents and Long Sessions
The most dramatic wins come from the workloads where context is large and reused constantly. This is why caching and the agent boom arrived together. A long-running agent carries a heavy payload on every turn: its system instructions, its growing memory bank, the schemas of every tool it can call, and the digest of everything it has learned so far. Without caching, that payload is re-billed at full price dozens or hundreds of times per task. With a well-placed cache breakpoint, the agent pays the discounted rate for the bulk of its context on every step after the first.
Early in the year, one public case study after another showed agent teams cutting input costs by 60 to 90 percent simply by restructuring how they assembled the context: write the stable parts once, select only the relevant retrieved passages, compress long histories into summaries, and isolate different agents with separate cached contexts so they do not trample each other's prefixes. These four moves — write, select, compress, isolate — were formalized by LangChain in its context-engineering framework, and they map perfectly onto cache design.
Model-Specific Tactics That Actually Matter
Not all caches are created equal, and the provider differences are worth knowing before you architect anything:
- Anthropic: explicit cache breakpoints with 75 percent cheaper cache reads on the newest models; cache entries have a time-to-live, so long-running sessions should refresh the cached region periodically rather than letting it expire.
- OpenAI: automatic prefix caching with no code changes required, but the discount varies by model tier, and the prefix must still be byte-identical; for agent loops, keep the assembly deterministic.
- Google Gemini: explicit caching with a minimum cacheable duration, which suits long-lived system prompts and shared knowledge bases that outlive individual sessions.
Whichever provider you use, the discipline is the same: measure, do not guess. Track cache hit rate, cached-token counts, and cost per request in your observability stack, the way you would track any other performance metric. Teams that instrument their prompts discover two things within a week: how often the cache is missing when it should hit, and how much money a single misplaced dynamic field was costing them.
What This Means for the "Is Prompt Engineering Dead?" Debate
Critics love to point out that tools like DSPy and OPRO can now discover prompts algorithmically, which is true and useful. But those tools optimize wording; they do not optimize structure. A meta-optimizer cannot fix a prompt assembly that interleaves volatile data with static scaffolding, because the problem lives in the serialization layer, not in the words. The discipline of cache-first design — deciding what the model sees, in what order, at what cost, with what caching behavior — is precisely the kind of work that remains stubbornly human. It is the difference between prompt engineering as a dying art and context engineering as a growing one.
The Bottom Line
Cache-first prompt design is the highest-leverage cost optimization available to LLM developers in 2026, and it is almost free to adopt: reorder your prompt assembly, freeze your stable prefix, mark the cache point, and instrument the hit rate. For teams running agents, the savings routinely exceed 70 percent of input spend — which, for many products, is the difference between a demo that bleeds money and a business that scales. The cheapest tokens really are the ones you already paid for.
