zerfoo

zerfoo

Pure Go ML framework -- inference, training, and serving. Embed any GGUF model in your Go application with go build ./....

241 tok/s on Gemma 3 1B Q4_K_M -- up to 28% faster than Ollama. Faster on all 4 benchmarked models. Zero CGo. 41 model architectures (25 families). Run models larger than RAM via memory-mapped I/O (229B MiniMax-M2 on 128 GB). EAGLE speculative decoding with built-in head training, QuaRot quantization, Q4_K fused GEMV (14x faster), Multi-LoRA serving, BitNet ternary inference. CUDA graph capture, Apple Metal kernels. Time-series training 4.6x faster with CUDA graphs. Tabular ML and time-series forecasting built in.

Benchmarks

Decode throughput comparison against Ollama on NVIDIA DGX Spark GB10 (Grace Blackwell, sm_121, 128 GB LPDDR5x).

Faster than Ollama on all models. Up to 28% faster on small models, 2% faster at 7B.

Memory-Mapped Model Loading

Zerfoo memory-maps GGUF files by default — no flags, no configuration. The entire file (or all shards of a split GGUF) is mmap'd via syscall.Mmap. Tensor data stays on disk and is paged into RAM on demand by the OS. Split GGUF files (multiple shards) are detected and mapped automatically from any shard path.

Results on DGX Spark (128 GB RAM, CPU-only):

// 128.8 GB model across 3 shards on a 128 GB machine.
// 809 tensors mapped. No heap allocation for weights.
m, _ := zerfoo.Load("./MiniMax-M2-Q4_K_M-00001-of-00003.gguf")
defer m.Close()
result, _ := m.Generate(ctx, "The meaning of life is")
// → "a priori is something"

Startup maps all shards and parses tensor metadata — no weight data is read until inference. The OS pages 128.8 GB of Q4_K_M quantized weights from NVMe as each matrix multiply streams through its superblocks. Ollama returns a 500 error on the same model on the same hardware.

Advanced Inference Features

EAGLE Speculative Decoding

Self-speculative decoding using a lightweight prediction head — no draft model needed. Based on EAGLE-3.

m, _ := zerfoo.Load("google/gemma-3-1b")
defer m.Close()
result, _ := m.Generate(ctx, "Explain quantum computing.",
    zerfoo.WithEAGLE("eagle-head.gguf"),
)

Train your own EAGLE head:

zerfoo eagle-train --model model.gguf --corpus data.txt --output eagle-head.gguf --epochs 5

QuaRot Weight Fusion

Hadamard rotation fused into weights at load time for uniform 4-bit quantization. Based on QuaRot.

zerfoo run --quarot model.gguf "Hello world"

Quantized KV Cache

Reduce KV cache memory by 6-7x with Q4 or Q3 quantization:

result, _ := m.Generate(ctx, prompt,
    zerfoo.WithKVDtype("q4"),  // 7.5x memory reduction
)

Tiered KV Cache

Automatically spill KV cache across three storage tiers as sequences grow — no OOM, no manual tuning:

• Hot: uncompressed tensors in GPU/CPU memory (recent tokens)
• Warm: compressed in CPU memory via block quantization
• Cold: serialized to disk as binary files (oldest tokens)

Layers are promoted and demoted automatically based on access frequency. Async prefetch moves cold layers back to hot before the decoder needs them.

result, _ := m.Generate(ctx, prompt,
    zerfoo.WithTieredKV(generate.TieredKVStoreConfig{
        ChunkSize:        64,  // warm-tier compression chunk size
        DemoteThreshold:  2,   // demote layers accessed < 2 times
        PromoteThreshold: 8,   // promote layers accessed > 8 times
        // ColdDir: "/var/cache/kv" // optional: persist cold tier across calls
    }),
)

Enable it on the Model API via zerfoo.WithTieredKV (wraps generate.WithTieredKV). Useful for long-context inference where the KV cache exceeds GPU memory.

TransMLA — MHA-to-MLA Conversion

Convert any MHA/GQA model to Multi-Head Latent Attention via SVD decomposition. Reduces KV cache by 80%+. Based on TransMLA.

zerfoo transmla --rank 512 --input model.gguf --output model-mla.gguf

Multi-LoRA Serving

Serve multiple LoRA adapters from a single base model. Per-request adapter selection via the OpenAI-compatible API:

curl http://localhost:8080/v1/chat/completions \
  -d '{"model": "gemma3-1b:my-lora", "messages": [{"role": "user", "content": "Hello"}]}'

BitNet Ternary Inference

Native support for ternary weight models ({-1, 0, 1}) where matrix multiplication becomes integer addition/subtraction. Based on BitNet b1.58.

Native Sparse Attention (NSA)

Hardware-aligned three-path sparse attention: coarse compression, fine-grained selection, and sliding window. Fused CUDA kernel. Based on NSA.

Hybrid CPU/GPU MoE

Place shared MoE experts on GPU, offload routed experts to CPU with SIMD kernels. Predictive prefetching achieves 98% hit rate. Based on KTransformers.

Audio Transcription

Transcribe WAV audio to text using Whisper or Voxtral speech-to-text models. Audio is chunked into 30-second segments, mel spectrograms are extracted, and each chunk is decoded in parallel:

wavData, _ := os.ReadFile("speech.wav")

m, _ := zerfoo.Load("openai/whisper-large-v3")
defer m.Close()

text, err := m.Transcribe(context.Background(), wavData)
fmt.Println(text)

Supports 16 kHz mono WAV input. Whisper uses 80 mel bins; Voxtral uses 128. Long audio is automatically chunked into 30-second segments and concatenated.

Quick Start

m, _ := zerfoo.Load("google/gemma-3-4b")  // downloads from HuggingFace
defer m.Close()
response, _ := m.Chat("Explain Go interfaces in one sentence.")
fmt.Println(response)

Installation

go get github.com/zerfoo/zerfoo

HuggingFace Download

Load accepts HuggingFace model IDs. Models are downloaded and cached automatically:

// Download by repo ID (defaults to Q4_K_M quantization)
m, err := zerfoo.Load("google/gemma-3-4b")

// Specify a quantization variant
m, err := zerfoo.Load("google/gemma-3-4b/Q8_0")

// Or load a local GGUF file
m, err := zerfoo.Load("./models/gemma-3-1b.gguf")

Streaming

Stream tokens as they are generated via a channel:

m, _ := zerfoo.Load("google/gemma-3-4b")
defer m.Close()

ch, err := m.ChatStream(context.Background(), "Tell me a joke.")
if err != nil {
    log.Fatal(err)
}
for tok := range ch {
    if !tok.Done {
        fmt.Print(tok.Text)
    }
}
fmt.Println()

Embeddings

Extract L2-normalized embeddings and compute similarity:

m, _ := zerfoo.Load("google/gemma-3-4b")
defer m.Close()

embeddings, _ := m.Embed([]string{
    "Go is a statically typed language.",
    "Rust has a borrow checker.",
})
score := embeddings[0].CosineSimilarity(embeddings[1])
fmt.Printf("similarity: %.4f\n", score)

Structured Output

Constrain model output to valid JSON matching a schema:

import "github.com/zerfoo/zerfoo/generate/grammar"

m, _ := zerfoo.Load("google/gemma-3-4b")
defer m.Close()

schema := grammar.JSONSchema{
    Type: "object",
    Properties: map[string]*grammar.JSONSchema{
        "name": {Type: "string"},
        "age":  {Type: "number"},
    },
    Required: []string{"name", "age"},
}

result, _ := m.Generate(context.Background(),
    "Generate a person named Alice who is 30.",
    zerfoo.WithSchema(schema),
)
fmt.Println(result.Text) // {"name": "Alice", "age": 30}

Tool Calling

Detect tool/function calls in model output (OpenAI-compatible):

import "github.com/zerfoo/zerfoo/serve"

m, _ := zerfoo.Load("google/gemma-3-4b")
defer m.Close()

tools := []serve.Tool{{
    Type: "function",
    Function: serve.ToolFunction{
        Name:        "get_weather",
        Description: "Get the current weather for a city",
        Parameters:  json.RawMessage(`{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}`),
    },
}}

result, _ := m.Generate(context.Background(),
    "What is the weather in Paris?",
    zerfoo.WithTools(tools...),
)

for _, tc := range result.ToolCalls {
    fmt.Printf("call %s(%s)\n", tc.FunctionName, tc.Arguments)
}

Supported Models

LLM Inference (41 architectures, 25 model families)

New architectures are auto-detected from GGUF metadata.

Tabular ML

Time-Series Forecasting

IBM Granite Time Series

Granite Time Series models are converted from HuggingFace using granite2gguf (part of zonnx). Supported tasks: forecasting, anomaly detection, classification, imputation, and embedding extraction.

Training

Train tabular and time-series models with built-in AdamW, learning rate schedulers, and early stopping:

import "github.com/zerfoo/zerfoo/tabular"

model := tabular.NewEnsemble[float32](engine, tabular.EnsembleConfig{
    InputDim:  10,
    OutputDim: 1,
    Models:    3,
})
trainer := tabular.NewTrainer(model, engine, tabular.TrainerConfig{
    LR:     0.001,
    Epochs: 50,
})
trainer.Fit(ctx, trainX, trainY)
predictions, _ := model.Predict(ctx, testX)

Fused SDPA graph node

layers/attention.FusedSDPA[T] wraps the existing ScaledDotProductAttention as a graph.Node[T] so callers can compose fused scale + softmax + matmul attention inside autograd graphs without re-implementing the math:

import (
    "github.com/zerfoo/zerfoo/layers/attention"
)

// Causal (decoder) SDPA, head_dim=64.
sdpa := attention.NewFusedSDPA[float32](engine, 64)

// Bidirectional (encoder) SDPA with explicit Q/KV head counts.
enc := attention.NewFusedSDPA[float32](engine, 64,
    attention.WithFusedSDPABidirectional[float32](),
    attention.WithFusedSDPAHeadCounts[float32](8, 8),
)

// Forward accepts (Q, K, V) or (Q, K, V, mask); Backward returns gradients
// for [Q, K, V] (and a nil slot for mask when one was supplied).
out, err := sdpa.Forward(ctx, q, k, v)

The node is numerically equivalent to the unfused Q @ Kᵀ → scale → softmax → dropout → V chain (fp32 ≤ 1e-5 fwd / 1e-5 bwd, fp64 ≤ 1e-12 fwd / 1e-10 bwd; see layers/attention/fused_sdpa_node_test.go).

CLI

go install github.com/zerfoo/zerfoo/cmd/zerfoo@latest

Examples

See the examples/ directory for runnable programs:

• chat -- interactive chatbot CLI
• rag -- retrieval-augmented generation with embeddings
• json-output -- grammar-guided structured JSON output
• embedding -- embed inference in an HTTP server
• api-server -- standalone API server
• inference -- basic text generation
• streaming -- token streaming
• fine-tuning -- LoRA fine-tuning
• automl -- automated model selection
• timeseries -- time-series forecasting
• distributed-training -- multi-node training
• agentic-tool-use -- function calling agent
• audio-transcription -- Whisper transcription

Documentation

Full documentation at zerfoo.feza.ai/docs/

• Getting Started -- install, pull a model, run inference
• Tutorials -- step-by-step guides
• API Reference -- generate, inference, serve APIs
• Cookbooks -- 12 runnable code recipes
• Architecture -- GPU setup, architecture overview
• Benchmarks -- throughput numbers
• Blog -- development updates and deep dives
• CONTRIBUTING.md -- how to contribute

License

Apache 2.0