Unified KV Cache Compression Methods for Auto-Regressive Models
-
Updated
Jan 4, 2025 - Python
#
kv-cache-compression
Here are 31 public repositories matching this topic...
LLM KV cache compression made easy
python transformers inference pytorch kv-cache large-language-models llm long-context kv-cache-compression
-
Updated
May 5, 2026 - Python
Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.
-
Updated
Mar 3, 2025
[NeurIPS'25 Oral] Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in latency (Qwen3/2.5, Gemma3, LLaMA3)
-
Updated
Feb 11, 2026 - Python
Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
-
Updated
Apr 13, 2025 - Python
[ICLR 2025] Palu: Compressing KV-Cache with Low-Rank Projection
-
Updated
Feb 20, 2025 - Python
Pytorch implementation for "Compressed Context Memory For Online Language Model Interaction" (ICLR'24)
-
Updated
Apr 18, 2024 - Python
First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.
machine-learning compression deep-learning pytorch transformer attention quantization iclr vector-quantization memory-optimization kv-cache google-research llm vllm llm-inference kv-cache-compression
-
Updated
Apr 17, 2026 - Python
This is the official repo of "QuickLLaMA: Query-aware Inference Acceleration for Large Language Models"
-
Updated
Jul 16, 2024 - Python
xKV: Cross-Layer SVD for KV-Cache Compression
-
Updated
Nov 30, 2025 - Python
(ACL2025 oral) SCOPE: Optimizing KV Cache Compression in Long-context Generation
-
Updated
May 28, 2025 - Jupyter Notebook
Accurate and fast KV cache compression with a gating mechanism
-
Updated
Apr 5, 2026 - Python
First open-source KVTC implementation (NVIDIA, ICLR 2026) -- 8-32x KV cache compression via PCA + adaptive quantization + entropy coding
compression pytorch nvidia transformer pca attention dynamic-programming quantization deflate entropy-coding memory-optimization kv-cache llm llm-inference kv-cache-compression iclr-2026
-
Updated
Apr 17, 2026 - Python
Native Windows build of vLLM 0.19.1 — no WSL, no Docker. Pre-built wheels + 34-file Windows patch + Multi-TurboQuant KV cache compression (6 methods, 2x cache capacity). PyTorch 2.10 + CUDA 12.6 + Triton + Flash-Attention 2.
windows gpu cuda pytorch nvidia triton msvc quantization kv-cache awq llm llm-serving vllm llm-inference flash-attention qwen kv-cache-compression turboquant vllm-windows multi-turboquant
-
Updated
Apr 26, 2026 - Python
Discrete Kakeya cover for LLM KV cache: D4/E8 nested-lattice quantisation realising a Kakeya-style tube-cover over the direction sphere. 2.4x-2.8x compression at <1% perplexity loss on Qwen3, Llama-3, DeepSeek, GLM-4, Gemma. Drop-in transformers.DynamicCache. pip install kakeyalattice.
transformers quantization discrete-geometry kv-cache long-context vllm llm-inference kv-cache-compression qwen3 lattice-quantization e8-lattice d4-lattice kakeya kakeya-set
-
Updated
Apr 30, 2026 - Python
LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
-
Updated
Sep 17, 2025 - Python
AI agent skill implementing Google's TurboQuant compression algorithm (ICLR 2026) — 6x KV cache memory reduction, 8x speedup, zero accuracy loss. Compatible with Claude Code, Codex CLI, and all Agent Skills-compatible tools.
python deep-learning transformer quantization codex vector-quantization model-compression inference-optimization kv-cache ai-agent llm llm-compression kv-cache-compression claude-code coding-agent iclr-2026 agent-skill turboquant polarquant qjl
-
Updated
Mar 28, 2026 - Python
Research and training stack for AVA — a tool-using, memory-aware virtual assistant targeting 4 GB VRAM. Spans custom transformers, verifier-RL, external memory, multi-domain benchmarks, and Gemma 4 inference optimization.
ai reproducible-research benchmarks personal-assistant quantization external-memory gemma virtual-assistant distillation inference-optimization tool-use edge-inference int4 llm long-context small-language-models kv-cache-compression recurrent-depth 4gb-vram verifier-rl
-
Updated
Apr 7, 2026 - Python
Near-optimal vector quantization for LLM KV cache compression. Python implementation of TurboQuant (ICLR 2026) — PolarQuant + QJL for 3-bit quantization with minimal accuracy loss and up to 8x memory reduction.
python deep-learning numpy transformer attention quantization vector-quantization model-compression inference-optimization memory-optimization kv-cache post-training-quantization llm llm-inference llm-compression kv-cache-compression iclr-2026 turboquant polarquant qjl
-
Updated
Mar 28, 2026 - Python
Repository for the paper: https://arxiv.org/abs/2510.00231
-
Updated
Oct 6, 2025 - Python
Improve this page
Add a description, image, and links to the kv-cache-compression topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the kv-cache-compression topic, visit your repo's landing page and select "manage topics."