Author: João Felipe De Souza

Empirical profiling of decode-path batching in autoregressive transformers.

Measures how batch size affects throughput, per-step latency, GPU memory, and batching efficiency during the decode (token generation) phase — the dominant cost in LLM serving.

For detailed architecture and design rationale, see DESIGN.md.

In LLM serving, the decode phase generates tokens one at a time per request. But multiple requests can share a single forward pass. The key tradeoffs:

• Small batch: GPU underutilized, low throughput, low latency
• Large batch: high throughput, but latency per step increases
• Too large: memory-bandwidth bound, throughput saturates or drops

Finding the sweet spot is critical for production serving systems.

For each (model, context_length, batch_size):

1. Prefill: process the full prompt in one forward pass, producing KV-cache
2. Decode: generate 64 tokens autoregressively using cached KV
3. Measure: prefill time, per-step decode latency, total throughput, peak memory

• Context lengths: 128, 512, 960
• Batch sizes: 1, 2, 4, 8, 16, 32, 64
• Total: 42 benchmark runs (21 per model)
• Key points repeated 3x for statistical confidence

• CUDA events for precise GPU timing
• 16 warmup steps before measurement
• 64 measured decode steps per configuration

See DESIGN.md for detailed rationale on timing, warmup, position limits, and OOM handling.

GPT-2, ctx=128:

Step latency stays nearly flat from bs=1 to bs=32. The GPU has spare compute capacity that batching fills for free.

Longer context means larger KV-cache per request, which fills memory bandwidth earlier and forces the sweet spot to a smaller batch.

GPT-2-medium, ctx=960:

At bs=64, throughput drops below bs=1 because peak memory (12.76 GB) exceeds physical VRAM (8.6 GB), causing implicit swapping.

The crossover point where step latency begins rising sharply:

This crossover is reproducible — repeated measurements show coefficient of variation below 5% at all key points.

Efficiency = actual throughput / (batch_size x bs=1 throughput):

At gpt2-medium ctx=960 bs=64, each request gets only 1.4% of its solo throughput — the GPU is completely memory-bound.

Measured peak memory vs analytical KV-cache estimate:

The analytical estimate underpredicts by 2-4x because it does not account for activation memory, framework overhead, and temporary buffers.

Latency-aware sweet spots deliver 90-97% of peak throughput at roughly half the per-step latency:

Key benchmark points were repeated 3x. Run-to-run variance:

Single-run vs repeated mean difference: within ±5% for all points, majority within ±2%.

decode-batching-profiler/
├── decode_benchmark.py          # Main benchmark sweep
├── plot_decode_benchmark.py     # Basic plots and summary
├── advanced_analysis.py         # Pareto, regime, memory, roofline
├── repeat_key_points.py         # Statistical confidence (3x repeats)
├── README.md
├── DESIGN.md
├── LICENSE
├── requirements.txt
├── results/
└── plots/

python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

python3 decode_benchmark.py \
    --models gpt2 \
    --ctx-lens 128 512 960 \
    --batch-sizes 1 2 4 8 16 32 64

python3 decode_benchmark.py \
    --models gpt2-medium \
    --ctx-lens 128 512 960 \
    --batch-sizes 1 2 4 8 16 32 64

python3 plot_decode_benchmark.py

python3 advanced_analysis.py

python3 repeat_key_points.py

• GPT-2 max_positions=1024; context capped at 960 to fit 64 decode steps
• Single-prompt workload; real serving has variable prompt lengths
• Results specific to RTX 2070; different GPUs have different crossover points
• No continuous batching; all requests start and end together
• Same prompt replicated across batch; real workloads have heterogeneous lengths

• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (2022)
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)