LLM caching is not normal caching, and teams that treat it as "Redis with extra steps" either cache nothing (paying full price forever) or cache the wrong thing (serving a confidently wrong answer to a different user). The short version up front: exact caching pays for itself immediately, semantic caching only makes sense layered on top of exact caching, and if you cannot measure your wrong-hit rate, the semantic layer is on probation. This essay is the playbook I use, drawn from two systems I have shipped: OnRamp, where every model call goes through a free-tier-first router with exact and semantic Redis caches behind it, and Estate360, a multi-tenant real estate CRM where an exact cache carries about a one hour TTL, semantic hits live for 24 hours, and semantic serving requires a 0.85 similarity threshold.
Projects: OnRamp and Estate360.
Why LLM caching is different
Three properties make LLM workloads unusually cacheable. Outputs are deterministic-ish (especially at temperature zero or near it). Real traffic is not a smooth distribution; it is a spike of near-identical requests. And the requests that repeat are exactly the expensive ones: support answers, document Q and A, onboarding checklists, the same questions rephrased by every new user. If your traffic looks like "what does this error mean", "how do I connect X", or "summarize this policy", the repeat rate is high enough that caching moves the bill, not just the dashboard.
What breaks the analogy with HTTP caching is correctness. Serving a stale image is mildly annoying; serving a stale or near-miss LLM answer can be actively wrong, confidently delivered. That is why the whole design is shaped around one question: how wrong can a hit be, and how would we know?
Exact caching: the free money
Exact caching is trivially safe and almost always worth doing first. Hash the normalized prompt plus the model plus the temperature, and serve the stored completion on a hash match. Normalization means trimming whitespace, collapsing case where your domain allows it, and pinning everything else. When the hash matches, the request is the same request, and the answer is the answer the model would (probably) give again.
- Safety: zero risk of wrong-answer injection. A hit is a bitwise repeat of a request already served.
- Hit rate: enormous on FAQ-shaped traffic and document Q and A; smaller on long, personalized generations.
- TTL by volatility: Estate360 keeps exact matches around for about an hour. Volatile content (prices, inventory, statuses) should expire fast; stable content (policy text, architecture docs) can live longer.
- Invalidation is key-scoped: change the prompt template and the hashes change; old entries rot naturally instead of poisoning new traffic.
Normalization deserves real care, because it decides what "the same request" means. Trim and collapse whitespace, unify quote and dash characters, canonicalize casing for case-insensitive domains, strip volatile metadata (timestamps, session ids, request ids) that would otherwise guarantee a miss. Keep the normalization versioned alongside the template version: change the normalizer and every old key becomes unreachable. The failure mode of sloppy normalization is silent 0% hit rate with all the machinery still running, which is the most expensive way to cache nothing.
Ship exact caching behind a flag, log every hit and miss, and let it run. It is the highest certainty dollar in the whole cost-reduction stack.
Semantic caching: real gains, real failure mode
Semantic caching serves the nearest prior answer to a rephrased request. Both systems I run do this with hashed n-gram cosine similarity over prompts rather than full embeddings; the mechanism is cheap, local and good enough to catch "how do I reset my password" versus "password reset steps". Semantic hits in Estate360 are kept for 24 hours and require a 0.85 similarity threshold. That number is not decoration. At 0.7 you serve answers to questions nobody asked; at 0.85 you mostly serve the same question asked politely and impolitely.
| Property | Exact cache | Semantic cache |
|---|---|---|
| Match rule | Hash of normalized prompt + model + temperature | Nearest neighbor above 0.85 cosine similarity over hashed n-grams |
| Typical TTL (Estate360) | About 1 hour | 24 hours |
| Failure mode | Practically none; misses are just misses | Near-miss answer served confidently to a materially different question |
| Best for | FAQ-shaped traffic, repeated templates | Rephrasings of the same underlying question |
The one rule I will not bend: semantic caching only makes sense on top of exact caching, never instead of it. Exact is your correctness floor; semantic is an optimization you add after the floor exists and you can measure above it.
Invalidation: where caches go to lie
A cache without an invalidation story is a wrong-answer generator with good uptime. My rules:
- Prompt template changes bust the cache. Version your templates and fold the version into the key. The day you tweak the system prompt, every cached answer under the old version is suspect.
- TTLs follow content volatility, not vibes. Onboarding checklist: long. Support macro about a pricing page that changes quarterly: short. Estate360 splits 1h exact and 24h semantic as a sane default, not a law.
- Never cache tool side effects. Caching the model's text is fine; caching the act of "book the site visit" or "send the reminder" is not a cache, it is a skipped action with plausible text attached.
- Never share personal data across users or tenants. Cache keys must include the tenant scope where the prompt can carry tenant data. OnRamp's onboarding answers can be shared across repos; Estate360's answers about a customer's units cannot be shared across workspaces, ever.
- Model and temperature are part of the key. Swap the model without busting keys and you are serving the old model's habits under the new model's name.
Rollout order, so you do not cache yourself into a corner
Ship this in the order that keeps blast radius small. First, exact caching behind a feature flag, hit and miss logged, for one workflow at a time; support answers or document Q and A are ideal. Second, watch the hit rate and the cost curve for a couple of weeks; if exact caching is not moving the needle on your traffic mix, semantic will not save you. Third, add the semantic layer behind the same flag, with the 0.85 threshold and spot checks running from day one. Fourth, widen coverage tenant by tenant or workflow by workflow, and keep the off switch one flag away. Teams that flip semantic caching on globally on day one are the teams that quietly serve wrong answers and never find out.
Measuring: hit rate is a vanity metric until you add wrong-hit rate
Both systems ship cache hits as first-class telemetry, and so should yours. Three numbers, in order of importance:
- Wrong-hit rate: the percentage of semantic hits that were near-misses, estimated by spot-checking a sample of served hits against what the model would have produced fresh. If you cannot measure this, semantic caching is on probation, full stop.
- Cost saved: tokens avoided, priced per provider, trended weekly. This is the number your finance person cares about and your arguments to keep the cache alive.
- Hit rate: useful for capacity planning and for spotting traffic-mix shifts. A sudden drop means the product changed under the cache, not that the cache got worse.
Exact cache hits need no wrong-hit rate; they are repeats. Semantic hits need continuous spot checks, forever, because the input distribution drifts and 0.85 is a threshold, not a guarantee.
In practice I run the wrong-hit audit like a tax audit: a fixed sample of served semantic hits, re-answered fresh by the model, with a human or a strict grader marking each pair as same-answer, close-enough or wrong. The close-enough bucket is the dangerous one, because it feels fine in aggregate and poisons trust one user at a time. Track the wrong share weekly; if it creeps, raise the threshold or shrink the TTL before you debug anything else.
When to skip caching entirely
- Low traffic. Below some modest volume, the engineering and observability cost exceeds the tokens saved.
- Highly personalized prompts. If every prompt embeds the user's unique context, hit rates collapse and you pay cache-lookup overhead for nothing.
- Compliance-sensitive output. If a stale or near-miss answer creates regulatory exposure, the correct cache TTL is zero.
- No telemetry. If you cannot see hits, misses and wrong-hits, you are flying blind with a correctness risk attached.
The short version
Hash everything first, serve repeats for free, then add semantic serving behind a high threshold with spot checks and honest telemetry. Cache text, never actions; scope keys by tenant; bust on template changes. If your LLM bill is dominated by repeated questions, this is the cheapest real engineering you can do. If your traffic is all bespoke, skip it and spend the effort on prompt quality instead.
I build caching layers and cost-control stacks like this for production LLM products. Fixed scope from ₹25,000, fixed quote in writing, free 15-minute intro call: https://kunjshah.in/contact.