quant.cpp loads GGUF files from HuggingFace, but only some model architectures are fully wired through the inference path. This page tracks what works, what loads-but-fails, and how to pick a model.

# CLI quickstart
quantcpp run smollm2          # SmolLM2-1.7B (recommended)
quantcpp run smollm2:135m     # SmolLM2-135M (smoke test only)
quantcpp run llama3.2:1b      # smaller download, slower

# Python quickstart
from quantcpp import Model
m = Model.from_pretrained("SmolLM2-1.7B")
print(m.ask("What is gravity?"))

✅ = works · ⚠️ = loads but inference is unreliable · ❌ = load fails fast with a clear error (since 2026-04-12)

If you load an unsupported architecture, the loader now prints:

tq_load_gguf: ERROR — model architecture 'phi3' is not supported.
  Detected 0 self_attn layers and no DeltaNet weights.
  This usually means the model uses fused QKV projection
  (e.g., Phi-3 `attn_qkv`) which quant.cpp does not yet handle.
  See docs/supported_models.md for the architecture support matrix.

…and tq_load_gguf returns NULL, so callers can fail-fast instead of silently producing garbage tokens.

quant.cpp generates one token at a time. Every token requires a lm_head matmul of shape [hidden_dim, vocab_size]. For a typical 1B model with hidden_dim = 2048:

Llama-3.2-1B has fewer parameters (1.0B vs 1.7B) but its lm_head matmul is 2.6x bigger, and on CPU it dominates wall time. External user benchmarks on Apple M3 (8-core CPU, 16 GB RAM):

Take-away: when picking a model for an embedded / laptop scenario, vocab size is a better predictor of interactive latency than parameter count. Pick the smallest vocab that produces output you're happy with.

Phi-3 / Phi-3.5 uses fused weight tensors instead of llama-style separate ones:

The loader detects these by name, stores the raw quantized pointers in new fields (gguf_w_qkv, gguf_w_up_gate), and the forward path dispatches a single matmul into a temp buffer for each, then memcpy splits the result into the existing per-section state buffers.

Phi-3 also uses LongRoPE with two per-frequency-pair rescaling tables (rope_factors_short, rope_factors_long) and a separate attention magnitude factor (rope.scaling.attn_factor). These extend RoPE rotation from the original 4096-token training context out to 131K. The forward path picks the short or long table based on position, applies the rescaled rotation in NeoX-style layout (pairs are (q[i], q[i+half]), not (q[2i], q[2i+1])), and multiplies Q by attn_factor only when pos >= original_context_length.

Why NeoX-style for Phi-3 specifically: llama.cpp's GGUF converter pre-permutes separate attn_q/k/v tensors so the standard interleaved RoPE works for Llama-family models. The fused attn_qkv tensor is NOT permuted, so we have to apply rotation in its native NeoX form.

Phi-3.5-mini at the recommended Q4_K_M quantization clocks in at ~32K vocab + 3.8B params, which makes the lm_head matmul the fastest of any model in the registry — the best speed/quality combo quant.cpp ships.

For users who want more quality than SmolLM2-1.7B, here are the three main contenders in the 3B-4B class and when to pick each:

• "I want fastest response" → Phi-3.5-mini — 32K vocab = smallest lm_head, ~8 tok/s on M3
• "I want best text quality" → Qwen3.5-4B — highest benchmarks, 262K context, DeltaNet hybrid saves 75% KV memory (partial support, improving)
• "I want strong math and code" → Phi-4-mini — HumanEval 74.4, MATH 64.0 (needs testing in quant.cpp)
• "I need images/audio/video" → Gemma 4 E2B — only multimodal option at this size (partial support)
• "I need Korean/Chinese/Japanese" → Qwen3.5-4B — purpose-built CJK tokenizer

Smaller vocab = faster generation. A 3.8B model with 32K vocab can be faster than a 1B model with 128K vocab — tested and confirmed on Apple M3.

If you tried a model that's not in the matrix above, please open an issue with:

• The HuggingFace repo + filename
• The exact tq_load_gguf: log lines (including architecture = '...')
• The first ~50 generated tokens (so we can see whether it's garbage, partial garbage, or just wrong-language)

Don't include the model file itself — link to the HuggingFace page.