Two recipes for serving GLM-5.2 on DGX Spark (GB10) clusters:

→ recipes/4x-dspark-1m/ — the 4th Spark plus a re-architected stack takes the same hardware family from 256K → 1M context and ~4.4 → ~24.7 tok/s (5.6×), with zero pruning: the fourth node's memory turns on 4-way KV sharding (DCP4), a ring-buffered DSpark draft adds lossless speculative decode that stays flat to depth, and two comm-fusion compile passes claw back the multi-node NCCL tax. Full build/launch/tuning walkthrough plus a speculator-finetune pipeline (+25% acceptance on this quant) in the recipe README. As of 2026-07 this is, to our knowledge, the highest quality-preserving context×throughput point published for GB10 clusters.

Serve the GLM-5.2-NVFP4-REAP-469B model (753B → 469B REAP-pruned, NVFP4 quantized) across a cluster of three NVIDIA DGX Spark nodes using vLLM with pipeline parallelism. Kept as the minimal-hardware option (3 nodes, no switch needed with dual direct links) and for its much higher prefill; for everything else prefer the 4× recipe above.

• Context: 256K tokens
• Throughput: ~4.4 tok/s decode, ~2,500–3,800 tok/s prefill
• Architecture: PP=3, TP=1, Ray distributed backend
• Quantization: NVFP4 (4-bit) with fp8 KV cache

┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│  Spark #1   │      │  Spark #2   │      │  Spark #3   │
│ (PP rank 0) │ ───▶ │ (PP rank 1) │ ───▶ │ (PP rank 2) │
│  26 layers  │      │  27 layers  │ NCCL │  26 layers  │
│  83.9 GiB   │      │  90.3 GiB   │      │  92.1 GiB   │
│  Head node  │      │   Worker    │      │   Worker    │
└─────────────┘      └─────────────┘      └─────────────┘
       │
       ▼
  :8000 HTTP API

Build or pull the vLLM Docker image for DGX Spark:

# The image must include vLLM, Ray, FlashInfer, and CUTLASS
# Target version: vLLM 0.1.dev16581+gdda4668b5
docker pull <your-registry>/vllm-node-dsv4-cl:latest

Download the model to every node at the same path:

# On each node:
mkdir -p ~/models
huggingface-cli download 0xSero/GLM-5.2-NVFP4-REAP-469B \
  --local-dir ~/models/GLM-5.2-NVFP4-REAP-469B

Disk space: ~287 GB per node. The head node (4 TB) is recommended.

Set up passwordless SSH from the head node to all worker nodes:

# On head node:
ssh-keygen -t ed25519
ssh-copy-id <user>@<worker-1-ip>
ssh-copy-id <user>@<worker-2-ip>

git clone https://github.com/bird/GLM-spark.git
cd GLM-spark

# Create .env from template
cp .env.example .env
# Edit .env with your node IPs and interface names
vim .env

Run these once on every DGX Spark node:

# 1. Kernel tuning (persistent across reboots)
sudo bash scripts/apply_sysctl.sh

# 2. Create swap (64 GB+ recommended)
sudo fallocate -l 64G /swapfile2
sudo chmod 600 /swapfile2 && sudo mkswap /swapfile2 && sudo swapon /swapfile2
echo '/swapfile2 none swap sw 0 0' | sudo tee -a /etc/fstab

# 3. Stop GUI and unnecessary services
sudo systemctl stop gdm snapd 2>/dev/null

# 4. Deploy daemons (OOM protector + cache dropper)
cp scripts/oom_fixer.sh scripts/cache_dropper.sh ~/
chmod +x ~/oom_fixer.sh ~/cache_dropper.sh
sudo systemd-run --unit=oom-fixer --working-directory=$HOME ~/oom_fixer.sh
sudo systemd-run --unit=cache-dropper --working-directory=$HOME ~/cache_dropper.sh

From the head node only:

export VLLM_SPARK_EXTRA_DOCKER_ARGS="--pid=host -v $HOME/models:/home/bird/models:ro"
nohup ./launch-cluster.sh recipes/glm-5.2-nvfp4-reap-469b.yaml -d > /tmp/recipe_launch.log 2>&1 &

Weight loading + MoE initialization takes approximately 15 minutes. Monitor progress:

# Inside the container on the head node:
docker exec vllm_ds4 tail -f /tmp/vllm_serve.log

When you see Application startup complete, the API is ready.

curl http://<head-ip>:8000/v1/models

curl http://<head-ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "GLM-5.2-NVFP4-REAP-469B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'

Four vLLM source patches are applied automatically at container startup:

• Disables DSA (Dynamic Shared Attention)
• Skips loading of indexer weights not present in REAP checkpoint
• Adds REAP expert remapping (original 256 → pruned 156 experts/layer)

• Adds posix_fadvise(DONTNEED) during safetensors loading to prevent page cache accumulation
• Critical on Grace Blackwell: unified memory means page cache competes with GPU for RAM
• Adds PP-aware file filtering (each node only loads its own checkpoint shards)

• Calls filter_files_by_pp_rank() to skip irrelevant shards during weight loading

• Forces num_stages=1 when BLOCK_DMODEL >= 512 (MLA with Lk=576)
• Fixes: OutOfResources: shared memory, Required: 102400, Hardware limit: 101376 on SM_121

Always use the cleanup script before relaunching — docker stop alone leaves orphaned Ray processes (due to --pid=host):

# On every node:
sudo bash ~/cleanup_ray.sh

Or from the head node:

# Clean all nodes at once:
ssh <worker-1> "sudo bash ~/cleanup_ray.sh"
ssh <worker-2> "sudo bash ~/cleanup_ray.sh"
sudo bash ~/cleanup_ray.sh

# vLLM server log (inside container on head node)
docker exec vllm_ds4 tail -f /tmp/vllm_serve.log

# Ray GCS server log
docker exec vllm_ds4 cat /tmp/ray/session_latest/logs/gcs_server.out | tail -20

# Per-worker Ray logs
ssh <worker-ip> "docker exec vllm_ds4 cat /tmp/ray/session_latest/logs/worker-*.err" | tail -20

If a worker node becomes unreachable (OOM crash):

1. Physically power-cycle the node (ASUS Ascent Sparks do not auto-recover)
2. Re-apply kernel tuning (persisted via sysctl.d, but verify):

   sudo sysctl -p /etc/sysctl.d/99-vllm-spark.conf

3. Restart daemons:

   sudo systemd-run --unit=oom-fixer --working-directory=$HOME ~/oom_fixer.sh
   sudo systemd-run --unit=cache-dropper --working-directory=$HOME ~/cache_dropper.sh

4. Clean Ray on all nodes, then relaunch

Cause: Ray GCS health check fails because the raylet is unresponsive during MoE init (memory pressure on Grace Blackwell).

Fix: Ensure RAY_health_check_failure_threshold=600 is set in launch-cluster.sh → get_env_flags(). With a 10-second period, this gives 100 minutes of grace.

Cause: Triton MLA decode kernel requires more shared memory than SM_121 provides (101,376 bytes).

Fix: The triton_decode_attention.py patch handles this. Verify it was applied by checking the startup log for [glm-5.2-patches] Patched triton_decode_attention.py.

Cause: Page cache from safetensors loading consumes all 121 GB of unified memory.

Fix:

1. Verify cache_dropper.sh is running (systemctl is-active cache-dropper)
2. Verify posix_fadvise patch is applied (check for [glm-5.2-patches] Patched weight_utils.py)
3. Ensure swap is configured (64 GB+ recommended)
4. Run apply_sysctl.sh for kernel tuning

Cause: Ray compiled graph timeout (default 600s, but MoE init can take 20+ minutes).

Fix: Set CONTAINER_RAY_CGRAPH_get_timeout=3600 in .env.

Cause: Node OOM'd. The system is alive at the network level but SSH daemon was killed.

Fix: Power-cycle the node physically. ASUS Ascent Sparks do not auto-power on after a crash.

Theoretical memory-bound decode limit: ~28.4 tok/s (273 GB/s ÷ 9.6 GB active params per token).

Actual: 4.4 tok/s = 15.5% of theoretical.

Overhead sources:

• --enforce-eager (no CUDA graphs): kernel launch overhead per step
• Python dispatch + scheduler overhead
• PP synchronization (3 NCCL hops per token)
• Socket transport (NCCL over TCP, no IB/RoCE)

Removing --enforce-eager would improve throughput significantly but causes heartbeat timeouts during CUDA graph compilation on Grace Blackwell.

Ngram speculative decoding was explored but does not work with PP=3 in this vLLM version due to multiple code bugs in the ngram + PP interaction path. The experimental patches are in experimental/ngram-patches/ for reference. See experimental/ngram-patches/ for details.

• vLLM — inference engine
• NVIDIA DGX Spark — hardware platform
• 0xSero — REAP-pruned NVFP4 checkpoint