How much does caching the system prompt KV state save — and when does the cache stop paying off?
Most production LLM deployments share a system prompt across all users. Without caching, every request pays the full prefill cost of that prompt. With caching, the KV state is computed once and reused by every subsequent request.
This benchmark isolates that gain: how much does system prompt caching reduce prefill work and TTFT, across system prompt sizes, user prompt sizes, output lengths, arrival rates, and competing prompt workloads.
For full methodology and design decisions see design.md.
This project closes a three-part KV reuse series:
| Project | Reuse scope |
|---|---|
| prefix-cache-sim | Between independent requests |
| multi-turn-kv-reuse-bench | Within a conversation |
| system-prompt-cache-bench | Across all users (shared system prompt) |
instances: 8
duration: 300s
cache policies: no_cache, LRU, LFU
system prompts: 128, 512, 1024, 2048, 4096 tokens
user prompts: 16, 64, 128, 256, 512 tokens
output lengths: 8, 32, 128 tokens
arrival rates: 1, 2, 4, 8 req/s
cache budgets: 256MB, 512MB
competing prompts: 1, 4, 8
popularity modes: uniform, zipf
| Strategy | TTFT mean | TTFT p99 | Hit rate | Prefill savings | TTFT reduction |
|---|---|---|---|---|---|
| LFU | 3465ms | 6765ms | 0.490 | 34.8% | mean -12.8% / p99 -12.5% |
| LRU | 3474ms | 6783ms | 0.485 | 34.4% | mean -12.6% / p99 -12.2% |
| no_cache | 3972ms | 7728ms | 0.000 | 0.0% | baseline |
| output_len_mean | prefill share of TTFT | LFU gain | LRU gain |
|---|---|---|---|
| 8 tokens | ~99.9% | 17.7% | 17.5% |
| 32 tokens | ~99.3% | 17.4% | 17.2% |
| 128 tokens | ~72.8% | 12.6% | 12.4% |
1. Prefill savings follow a closed-form ratio. savings_frac ~= system_prompt_len / (system_prompt_len + user_prompt_len) Theoretical prediction matches simulation results closely.
2. Final TTFT gain is smaller than prefill savings. With ~35% prefill-token savings, mean TTFT improves by ~13% overall. The gap is explained by decode and queueing time, which caching cannot reduce.
3. Cache is most valuable in short-output interactive workloads. At output_len=8 and 32, prefill accounts for ~99% of TTFT. Caching yields ~17.5% TTFT reduction in this regime.
4. Decode dilutes the visible TTFT gain as output length grows. At output_len=128, prefill share falls to ~73% and TTFT reduction drops to ~12.5%.
5. LFU is slightly better than LRU under skewed workloads. The difference is small. Cache sizing matters more than eviction policy.
6. Cache usefulness collapses when too many prompts compete for too little budget. The right sizing rule is based on the hot working set, not the total prompt count.
7. System prompt caching reduces prefill work by up to 99%. When the system prompt dominates total prompt length and the cache is warm.
High value:
system prompt is long
user prompt is relatively short
interactive latency matters
tool-using assistants, agents, fixed personas, large hidden prompts
Low value:
user prompts dominate total prompt size
outputs are very long and decode dominates total request time
prompt working set is too large for the available KV cache budget
results/
sweep_summary.csv
summary.txt
plots/
savings_vs_ratio.png
ttft_vs_output_len.png
hit_rate_vs_budget.png
competing_prompts.png
policy_comparison.png
system-prompt-cache-bench/
├── src/
│ ├── __init__.py
│ ├── config.py
│ ├── workload.py
│ ├── cache.py
│ ├── simulator.py
│ ├── metrics.py
│ └── analysis.py
├── results/
├── plots/
├── LICENSE
├── design.md
├── README.md
├── requirements.txt
└── run.py
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python run.pyMIT License -- Copyright (c) 2026 Joao Felipe De Souza