Event-driven benchmark of per-tenant KV-cache isolation policies for multi-tenant LLM serving, measuring SLO compliance, fairness, and stability under noisy neighbor and burst scenarios.

Author: Joao Felipe De Souza Year: 2026

Production LLM serving infrastructure is typically shared between multiple tenants with different SLO requirements and priorities. Without explicit isolation, a noisy neighbor can degrade the experience of premium tenants even when global capacity is sufficient.

This benchmark asks:

1. Which isolation policy best protects premium tenant SLOs?
2. Is there a policy that simultaneously protects all tenants?
3. What is the fairness cost of strong isolation?
4. Are priority-based reclaim policies stable under simultaneous bursts?

Three tenants compete for shared KV-cache capacity:

The central result is that isolation policy choice is a deliberate trade-off. Every policy protects a different subset of tenants.

Under noisy_neighbor workload, hard quota reduces tenant_A SLO violations from 82 percent with global FCFS to less than 1 percent. The cost is that tenant_B is left unprotected.

Priority reclaim meets tenant_B SLO in 75 percent of configurations. However, under bursty_mixed_tenants, p99 TTFT reaches 470000ms, indicating a reclaim loop that degrades all tenants simultaneously. This finding shows that priority reclaim requires a minimum capacity floor per tenant to be safe in production.

Soft quota borrowing maintains high Jain fairness but fails to protect tenant_B in most scenarios because borrowed capacity flows to tenant_C without sufficient control.

Global FCFS achieves zero SLO compliance for tenant_A and tenant_B across all tested configurations. It is the worst policy for any workload where SLOs must be maintained.

global_fcfs_no_isolation No per-tenant limits. Pure FCFS using the global KV budget. Baseline that demonstrates the absence of isolation.

hard_quota Each tenant has a dedicated KV allocation equal to its quota fraction. Cannot exceed quota even when other tenants are idle. Maximizes isolation, reduces utilization and fairness.

soft_quota_borrow Tenants can borrow idle capacity from others with a headroom reserve. Improves utilization while maintaining partial isolation.

priority_reclaim High-priority tenants can reclaim borrowed capacity from lower-priority tenants. Best cross-tenant balance but unstable under simultaneous bursts.

• balanced_3tenants: all tenants at moderate load
• noisy_neighbor: tenant_C bursts aggressively
• premium_latency_sensitive: tenant_A at high load with tight SLO
• long_context_background: tenant_C runs large background jobs
• bursty_mixed_tenants: all tenants burst simultaneously

Per-tenant: throughput_rps, p99_ttft_ms, slo_violation_rate, slo_met, borrow_events, reclaim_events, completed, rejected

Global: global_throughput_rps, jain_fairness_index, avg_kv_reserved_mb, total_borrowed_mb, total_reclaimed_mb

From the project directory:

cd ~/dev/multi-tenant-serving-bench
source venv/bin/activate
python -u run.py

results/summary.csv
results/tenant_detail.csv
plots/

multi-tenant-serving-bench/
|-- src/
|   |-- __init__.py
|   |-- config.py
|   |-- workload.py
|   |-- tenants.py
|   |-- scheduler.py
|   |-- simulator.py
|   |-- bench.py
|   |-- analysis.py
|-- results/
|-- plots/
|-- run.py
|-- SUMMARY.txt
|-- DESIGN.md
|-- LICENSE
|-- README.md
|-- requirements.txt
|-- .gitignore

This is a KV-allocation simulation, not a full serving stack. Known simplifications include no real GPU execution, no actual token generation, no batching at step level, no network or scheduling overhead, simplified reclaim without partial eviction, and no token streaming. These simplifications isolate the economics of per-tenant KV admission.

Multi-tenant LLM serving requires explicit per-tenant KV isolation. No single policy simultaneously satisfies all tenant SLOs. Hard quotas maximize isolation for premium tenants at the cost of fairness. Priority reclaim achieves better cross-tenant balance but requires minimum capacity floors to prevent instability under simultaneous bursts. Global FCFS is unsuitable for any workload where tenant SLOs must be maintained.

MIT License. See LICENSE.