Caching and Batching
Cut delay and compute with response caching, prompt caching, and batch jobs, and know when each one is unsafe.
Objective
By the end of this lesson, you can choose between response caching, prompt caching, and batching for a workload, and name the privacy and freshness risks of each.
Why this is an architecture concern
Every model call takes time and compute. For a person waiting on a phone with a weak signal, a 10-second reply feels broken. For a small organization, compute usage is a real, limited resource that partners will ask about. Efficient design keeps a system usable and sustainable.
Three terms first:
- Latency: how long a person waits for a response.
- Throughput: how many requests the system handles in a period.
- Tokens: the chunks of text models read and write. Usage, speed, and limits are usually measured in tokens.
Response caching
A response cache stores answers to questions you have already answered. When the same question comes again, you return the saved answer without calling the model.
It works well for shared, stable answers, like hours and locations. It is dangerous for:
- Personal answers. A cached answer about one client's case could be shown to another person. Never cache across users when the answer depends on who is asking.
- Changing facts. If hours change, a cached answer goes stale. Set a time to live (TTL): how long an answer stays valid. Clear the cache when source documents change.
Normalizing the question (lowercase, trimmed, extra spaces removed) raises the hit rate. Semantic caching matches questions with similar meaning using embeddings. It hits more often, and it can return an answer to a question that only looked similar. Test it carefully.
The first run downloads Python into your browser (several MB).
Prompt caching
Many providers offer prompt caching: if the start of your prompt (system prompt, long documents, tool definitions) is identical across requests, the provider reuses its work on that part. This can reduce delay and usage for long, repeated prompts.
To benefit, put stable content first and changing content (the user's question) last. Check your provider's documentation for how long cached prompts last and how they are stored.
Batching
Batching sends many requests together as one job, processed when capacity allows, often over hours. Providers often offer batch processing for work that does not need an instant answer.
Good for: tagging last month's intake notes, translating a resource library, running your full test set overnight.
Not for: anything a person is waiting on.
| Technique | Best for | Main risk |
|---|---|---|
| Response cache | Repeated, shared, stable questions | Leaking personal answers, stale facts |
| Semantic cache | High-volume FAQ traffic | Answering a different question |
| Prompt caching | Long, repeated system prompts and documents | Little, if stable parts come first |
| Batching | Offline bulk work | Slow by design |
Quiz: Caching check
Go deeper: Systems Administration, Simplified: A Practical Blueprint for Uptime, Security, and Scale
Discussion
Ask. Answer. Build together.
Loading…