Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
-
Updated
Feb 20, 2026 - Python
#
speculative-decoding
Here are 121 public repositories matching this topic...
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
retrieval chatbot rag habana large-language-model chatpdf llm-inference 4-bits speculative-decoding llm-cpu streamingllm intel-optimized-llamacpp neural-chat neural-chat-7b autoround gaudi3
-
Updated
Oct 8, 2024 - Python
Lucebox optimization hub: hand-tuned LLM inference, built for specific consumer hardware.
kernel cuda cuda-kernels nvidia-cuda luce rtx3090 llama-cpp local-ai qwen speculative-decoding dflash megakernel speculative-prefill pflash lucebox
-
Updated
May 8, 2026 - C++
Large-scale LLM inference engine
-
Updated
May 8, 2026 - C++
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
audio eagle quantization diffusion vlm llm qwen speculative-decoding llm-compression hunyuan deepseek fp4 dflash
-
Updated
May 7, 2026 - Python
scalable and robust tree-based speculative decoding algorithm
-
Updated
Jan 28, 2025 - Python
Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024
-
Updated
Apr 13, 2026 - Python
[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
-
Updated
Aug 31, 2024 - Python
Native MTP Speculative Decoding On Apple Silicon | 2x - 2.5x decode TPS increase at temp 0.6 | MLX-native, OpenAI API/Anthropic-compatible serving, no external drafter.
metal mtp mlx inference-engine apple-silicon local-ai qwen speculative-decoding speculative-sampling openai-compatible qwen3-next anthropic-compatible native-mtp mtplx
-
Updated
May 9, 2026 - Python
REST: Retrieval-Based Speculative Decoding, NAACL 2024
-
Updated
Mar 5, 2026 - C
LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving
machine-learning cuda inference pytorch transformer triton moe quantization language-model inference-engine kv-cache tensor-parallelism llm speculative-decoding pagedattention continuous-batching
-
Updated
Apr 24, 2026 - Python
Pure Rust Inference Engine
rust cuda transformers ssm mamba dgx openai-api llm-inference speculative-decoding gb10 nvfp4 dgx-spark
-
Updated
May 8, 2026 - Rust
Tree-based speculative decoding for Apple Silicon (MLX). ~10-15% faster than DFlash on code, ~1.5x over autoregressive. First MLX port with custom Metal kernels for hybrid model support.
-
Updated
Apr 15, 2026 - Python
LLM Inference on consumer devices
-
Updated
Mar 17, 2025 - Python
[ICML 2025] |TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation
-
Updated
May 19, 2025 - Python
Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.
-
Updated
Dec 2, 2024 - Python
[NeurIPS'23] Speculative Decoding with Big Little Decoder
-
Updated
Feb 6, 2024 - Python
llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP speculative decoding for ~30-50% throughput gains
metal cpp vulkan cuda quantization mtp gemma on-device-ai apple-silicon walsh-hadamard low-bit-quantization llama-cpp local-llm llm-inference gguf speculative-decoding kv-cache-compression gemma3 multi-token-prediction turboquant
-
Updated
May 8, 2026 - C++
vLLM patcher for Qwen3.6 on consumer NVIDIA — Qwen3.6-35B-A3B-FP8 (192 tok/s, +68% over stock) + Qwen3.6-27B-int4-AutoRound + 256K context. 126 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN streaming, structured boot summary, one-command installer, 1958 tests. v7.72.2.
cuda nvidia moe gdn ampere structured-output long-context fp8 vllm llm-inference qwen speculative-decoding tool-calling qwen3 rtx-3090 runtime-patches dflash turboquant ampere-sm86 rtx-a5000
-
Updated
May 5, 2026 - Python
[ICLR 2025] SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
-
Updated
Feb 21, 2025 - Python
Improve this page
Add a description, image, and links to the speculative-decoding topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the speculative-decoding topic, visit your repo's landing page and select "manage topics."