@bokiko benchmarked the feature on three hosts and found every setting is
either no faster or no longer coherent. Reproduced here on a 25 GB box, and
the numbers are worse than "a quality/speed tradeoff":

  - budget=8 -> hellaswag 30% vs 90% with it off (25% is the chance floor)
  - budget=4 -> decode is literal noise ("The **1...: s2151:")
  - MTP acceptance 0%. Which experts survive the cap depends on cache
    residency at the moment of the forward, so draft and verify do not
    compute the same function -- the invariant #294 just established for
    #163, violated through cache state instead of kernel dispatch. SPEC_PIN
    cannot fix that and should not try: it is feature semantics, not
    dispatch.
  - 0.13 tok/s vs 0.30 baseline, while loading 14.66 experts per layer
    against a topk=8 baseline. The cap does MORE disk I/O than not using it.
    "~335 GB I/O saved" counts dropped experts, not bytes not read.

EXPERT_BUDGET>0 is now ignored unless EXPERT_BUDGET_EXPERIMENTAL=1, with the
measurements printed. The code stays compiled and developable: MoE-Spec
(arXiv 2602.16052) is not a wrong idea, this implementation just has no
point where it is both faster and correct. Re-enabling it by default needs a
quality number next to every speed number.

Also gates the cap to decode (S<=4), @woolcoxm's fix from #292/#298: during
prefill the batch union is 30-100+ experts, and capping to 4-8 drops most of
them, corrupting the hidden state and therefore the KV cache. Necessary but
not sufficient -- the run above already includes it.

Removes issue_budget.md, issue_diskio.md and issue_grouped_quant.md: design
notes in the repo root, unlinked from any README, whose only user-facing
content was "EXPERT_BUDGET=6-8 -- good speedup, minimal quality loss" and
"+83% decode at budget=4". Measurement says otherwise, and they were the
only thing on main telling anyone to switch this on. They stay in history.

Reported-by: bokiko <#303>
Co-Authored-By: woolcoxm <#292>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The CUDA backend had no fmt=4 case anywhere. Every kernel assumed per-row
scales (one scale per output row). Grouped int4 (fmt=4) needs per-group
scales (one scale per gs=64 elements along the input dim), applied inline
during the dot-product accumulation, not once at the end.

Changes to backend_cuda.cu:
- ColiCudaTensor: added gs, ng fields (group size + groups per row)
- GroupDesc: added g_gs/g_ng, u_gs/u_ng, d_gs/d_ng for expert kernels
- row_bytes/weight_at: added fmt=4 case (same nibble layout as fmt=2)
- scale_at(): new device helper — per-row for fmt 1/2/3, per-group for fmt=4
- quant_matmul: scale applied inline via scale_at (not at end)
- grouped_hidden/down, grouped_hidden_w4/_dual, grouped_down_w4: same
- attention_absorb_kernel/batch_kernel: per-group scale in q/v projections
- All kernel launch sites updated to pass gs,ng
- coli_cuda_tensor_upload_grouped: new upload that accepts gs, allocates
  O*ng scales for fmt=4; old upload delegates with gs=0
- offset_to_signed_s4 conversion fires for fmt=4 (same encoding as fmt=2)

Changes to backend_cuda.h: declare coli_cuda_tensor_upload_grouped
Changes to backend_loader.c: resolve + wrap the new symbol
Changes to glm.c: qt_cuda_upload routes fmt=4 to grouped upload;
  dropped the w->fmt!=4 guard in matmul_qt_ex (CUDA now handles fmt=4)

Verified: SCORE mode (log-likelihood eval) with CUDA_DENSE+CUDA_ATTN
on g64 model — single request, 401-token request, no crash, correct scores.

Refs JustVugg#292 JustVugg#298

… PR JustVugg#298)

The fmt=4 CUDA support is implemented (commit 7499eac) but causes repeated
system crashes (0x116 VIDEO_TDR_FAILURE) on sm_120 Blackwell GPUs. Despite
kernel chunking and TDR registry changes, the crashes persist. Restoring the
guard keeps fmt=4 on the CPU path (matmul_i4_grouped) which is correct and
stable. The CUDA code remains for when the driver stability issue is resolved.

See PR JustVugg#298 for the full crash analysis and implementation details.

matmul_i4_grouped is the reference the CUDA fmt=4 port (#298) is expected to
reproduce, and it had no test of its own. @woolcoxm is currently debugging a
CUDA backend against an oracle nobody had verified, which is two moving
targets at once -- and he can't cross-check on CPU, since a 5-prompt run
takes 8 hours on the 744B model.

This checks matmul_i4_grouped against a plain-C reference that dequantizes
nibble -> (v-8)*scale[i/gs] and accumulates in double, over 11 shapes: I a
clean multiple of gs, a partial last group (the glen clamp), odd I (the
scalar nibble tail), gs > I, gs=16/64/128, S>1, and the nibble extremes
0x00/0xFF -- which decode to -8/+7 because the format is offset-encoded, not
two's complement. Reading that backwards turns 15 into -1 and looks like
data-dependent noise rather than a bug.

All 11 shapes match to ~1e-8 relative, so the CPU kernel is exact and can be
trusted as the reference.

One note on the tolerance, because the first draft of this test got it wrong
and "found" a bug that wasn't there: the error is compared against the sum of
|terms|, not against |result|. A dot product of signed terms can land near
zero through cancellation, and then a 1e-6 absolute error -- ordinary f32
accumulator precision -- reads as a 1e-3 relative one. A wrong scale index or
a wrong group boundary shifts the result by a fraction of the terms, so it is
still caught at 1e-6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The Metal GEMV shader (mm_gemv, backend_metal.mm) handled fmt 0-3 only.
Anything else -- including dev's grouped-int4 format (fmt=4: packed
int4 nibbles, same layout as fmt=2, but one f32 scale per `gs`-element
group along I instead of one scale per row; CPU reference
matmul_i4_grouped in quant.h, active CUDA tier in JustVugg#298/JustVugg#451) -- fell
through the shader's unconditional `else` arm and was decoded AS RAW
F32. This was a silent-wrong-answer defect, not just a missing-
acceleration gap: bind_gemv (used by the fused decode-attention
kernels, coli_metal_attn_decode/coli_metal_layer_decode) accepted a
per-weight fmt from its caller with no validity gate at all, so any
fmt=4 dense/attention tensor reaching that path would "succeed" (ok=1)
while silently computing garbage. coli_metal_matmul/coli_metal_gemm had
explicit fmt<=3 gates and so safely rejected fmt=4 (0/CPU-fallback);
bind_gemv had no such safety net.

COMMIT ORDERING (bisect-safety, see PR_BODY.md sec 1 for the full
argument): this commit lands FIRST, before the qt_from_disk allocator
fix (next commit). At this commit alone, the capability is CORRECT but
INERT: qt_from_disk's fmt=4 scale buffer is still allocated with a bare
falloc() (unregistered, not page-aligned), so resolve() in
backend_metal.mm can never find it, and every real fmt=4 dense/
attention tensor's scale buffer permanently misses bind_gemv/
coli_metal_gemm regardless of what this commit adds to the shader --
safe CPU fallback, not a wrong answer. No commit in this branch is ever
"the shader is fixed but a scale buffer resolves to the wrong/stale
allocation" -- fixed-but-unreachable, then reachable-because-fixed, in
that order, never the reverse. Verified empirically at this exact
commit (this branch, this session): both METAL=0/METAL=1 build clean,
0 warnings, and `make metal-test` passes 27/27 -- metal-test's fmt=4
cases construct their own tensors directly via posix_memalign +
coli_metal_register (never through qt_from_disk), so none of them
depend on the allocator fix landing first; the capability this commit
adds is exercised and proven correct here, just not yet reachable from
a real loaded model.

Shader: unlike fmt 1-3 (single `acc * scale[o]` after the full dot
product), fmt=4's group scale must be folded into the accumulation
per-group, before summing across groups -- this is matmul_i4_grouped's
actual math, not a simplification. New `constant int& gsz [[buffer(9)]]`
parameter carries the group size (ignored, harmlessly, for fmt!=4).
Each of the 32 SIMD lanes owns one packed byte (2 elements) per 64-wide
stride -- memory access stays coalesced and a group never splits a byte
(gs is always even, per the loader's candidate list). Deliberately not
vectorized like fmt=2's uchar4/float4 path -- see PR_BODY.md sec 7 for
why that tradeoff was made on purpose.

Host plumbing: gs threaded through every path that reaches mm_gemv --
coli_metal_matmul, coli_metal_gemm, and bind_gemv (-> AttnW ->
coli_metal_attn_decode / coli_metal_layer_decode) -- plus the colibri.c
call sites (matmul_qt_ex's Metal-GEMM gate now includes fmt==4;
attention_rows/layer_forward_rows pass .gs alongside each .fmt already
being passed, a no-op for every model not using fmt=4). fmt_bytes and
the new fmt_scale_bytes fix coli_metal_matmul's scale-buffer wrap size,
which was previously hardcoded to O floats (wrong for fmt=4's
O*ceil(I/gs)). kv_b (MLA absorption core) and the batched routed-expert
MoE path (moe_gemv/coli_metal_moe_block) are deliberately untouched --
both already gate fmt=4 to a safe CPU fallback today, and extending
them is a structurally separate kernel family; see PR_BODY.md sec 7 for
the full scope statement.

Tests (tests/test_backend_metal.mm): new cpu_ref_grouped (double-
precision oracle mirroring matmul_i4_grouped, same construction
tests/test_i4_grouped.c already established) and run_grouped harness,
using a magnitude-relative tolerance (justified in PR_BODY.md sec 5)
instead of the existing ymax-relative convention, since a grouped dot
product's cancellation can otherwise misreport as a kernel defect.
Covers the spec's shape matrix (I mult/non-mult of 64, I<=64
degenerate, S=1/S>1, outlier-heavy rows) via coli_metal_matmul, one
case via coli_metal_gemm (the large-batch path), and two via a new
run_attn_grouped harness that exercises the real fused-attention
bind_gemv path end-to-end (not just the standalone kernel entry point).
(These two attn cases have known blind spots as first committed here --
closed two commits later, see that commit's message and PR_BODY.md
sec 10; kept as originally written here for bisect fidelity.)

make metal-test: 15/15 (stock e9b3614) -> 27/27 (12 new fmt=4 cases;
original 15 unchanged -- the negative control). make glm METAL=0/
METAL=1: clean, 0 warnings, both before and after. make test-c /
test-python: unaffected, all pass. Full gates, evidence classes,
and DEVIATIONS/UNCERTAINTIES in PR_BODY.md (untracked, worktree root).

(cherry picked from commit f0fa5a9)
(cherry picked from commit 82984cdeef67ede71d48b32f2ddd4734f4622eba)

… semantics

The merged JustVugg#298 gave the CUDA kernels per-group scale handling for fmt=4 —
without the same semantics a g64 container on Vulkan would decode with per-row
scales (the exact bug JustVugg#298 fixed). All three shaders gain a fmt==4 branch:
int4 nibble decode with one scale per gs inputs, applied to the packed-word
partial (host gates gs to multiples of 8 so a word never straddles a group;
per-row scaling is skipped like fmt=5). Group size flows as an explicit
parameter through the upload-triggering entries into ColiVkTensor and the
push constants; engine call sites pass QT.gs and the VK gates accept fmt=4
via VK_FMT_OK (word-aligned gs only — anything else stays on the CPU path,
which JustVugg#298 already fixed).

Harness: ref helpers generalized to runtime group size (g_ref_gs); fmt=4
cases at gs=64 across the real shapes (dense, o-proj, batch, expert_group,
matmul_pair, absorb incl. S=2 causal + window) plus a gs=32 sanity case.

… semantics

The merged JustVugg#298 gave the CUDA kernels per-group scale handling for fmt=4 —
without the same semantics a g64 container on Vulkan would decode with per-row
scales (the exact bug JustVugg#298 fixed). All three shaders gain a fmt==4 branch:
int4 nibble decode with one scale per gs inputs, applied to the packed-word
partial (host gates gs to multiples of 8 so a word never straddles a group;
per-row scaling is skipped like fmt=5). Group size flows as an explicit
parameter through the upload-triggering entries into ColiVkTensor and the
push constants; engine call sites pass QT.gs and the VK gates accept fmt=4
via VK_FMT_OK (word-aligned gs only — anything else stays on the CPU path,
which JustVugg#298 already fixed).

Harness: ref helpers generalized to runtime group size (g_ref_gs); fmt=4
cases at gs=64 across the real shapes (dense, o-proj, batch, expert_group,
matmul_pair, absorb incl. S=2 causal + window) plus a gs=32 sanity case.

Retarget PR JustVugg#377 onto current dev (origin/dev @ 4aca059) after JustVugg#391 split
glm.c into colibri.c + quant.h/sample.h/kv_persist.h/telemetry.h. Reconciles
four collisions:

- JustVugg#391 split: all glm.c hunks resited -- prof_*/g_prof_io/ProfBase live in
  colibri.c (NOT telemetry.h); tiers_emit/emap_emit -> telemetry.h (with a
  rammap_slot forward-decl); st_fd_is_tmpfs/st_fd_fs_magic -> st.h; .coli_kv
  state-dir -> serve_ctx_init/run_serve/run_serve_mux + main.
- DISK-CLASS (1f00142): prof_physical_read_bytes/ProfPhysicalWire merged ON
  TOP of dev's dc_* fields; PROF protocol line extended 9->17 fields
  additively; expert_load_impl 6-arg `demand` signature preserved; g_prof_io
  routed through prof_ssd_tensor_bytes (tmpfs-excluded).
- DUAL-SSD mirror (JustVugg#298/JustVugg#469): ESlot.backing hand-merged with dev's
  aslab/afslab; map_of_fd exact-length + MADV_HUGEPAGE composes with rep_bfd.
- int3 fmt=5 (JustVugg#168): rammap_bind_one reuses dev's qt_resolve_fmt/detect_group_size
  (inline fmt-detection dropped; duplicate detect_group_size not re-added).

Verified: make colibri 0 warnings; make check 187 tests pass (test_uring skips
in sandboxes via the PR's helpers; test_rammap builds against colibri.c and
passes). Default path byte-identical -- oracle-safe by construction.

Co-Authored-By: Claude <noreply@anthropic.com>

… call sites repaired

Rebase of the HIP single-source backend onto dev after JustVugg#298. Findings:

- The per-group (fmt=4) kernel logic needs NO compat-layer work: backend_cuda.cu
  is compiled unchanged for both vendors, so JustVugg#298's fix flows through to ROCm
  automatically. The compat header required zero new mappings (all 33 CUDA
  runtime symbols dev uses were already covered). ROCm does not inherit the
  g64 garbage-output bug.
- One conflict, resolved as a union: our COLI_GPU_HAS_WMMA compile-gate now sits
  alongside JustVugg#431's row-threshold check on the W4A16 tensor-core branch.
- tests/test_backend_cuda.cu did not compile on dev: JustVugg#298 added the trailing gs
  parameter to coli_cuda_matmul and split tensor_upload/_g, but six matmul call
  sites and seven upload call sites were left on the old signatures. Repaired
  (gs=0 / _g variant). This is invisible to CI today because engine-cuda-syntax
  only syntax-checks backend_cuda.cu, never the test binary — the gpu-compile
  target in this PR does build it, which is how it surfaced.

Verified on RX 9070 XT (gfx1201, ROCm 7.2): full kernel suite incl. fmt=4
per-group coverage passes, 3/3 stable runs; engine builds; make check green.

…#298)

#298 fixed the dense-resident path's callers after they fed grouped-scale
tensors to kernels that read scales per row (CUDA_DENSE=1 produced garbage).
The entry points themselves still accept them:

  coli_cuda_matmul   uploads via coli_cuda_tensor_upload (no gs), so
                     scale_count collapses to O and the [O, ceil(I/gs)]
                     buffer is silently truncated, then quant_matmul reads
                     scales[o] per row.
  coli_cuda_expert_mlp  same kernels, same assumption.

Return 0 for fmt=4 at both, so a grouped tensor takes the expert-group path
(#451, which carries per-group scales) or stays on the CPU — a fallback, never
a wrong answer. The caller-side guard in colibri.c already excludes fmt=5 the
same way; this makes the ABI safe regardless of who calls it next.

make check 105/105, zero warnings.

…A entry points (#334/#298)

Verified locally on current dev: clean build, token-exact unchanged (fp 32/32, int4 21/32 — byte-identical / no regression).

…compat.h

Adds AMD GPU support through a single-source HIP/ROCm compat layer with a WMMA dispatch gate. Both merge gates met rigorously by @noobdev-ph: (1) rebased on current dev, CI 9/9 green, zero compat-layer churn; (2) validated on real AMD hardware (RX 9070 XT / gfx1201 RDNA4, ROCm 7.2) against a real fmt=4 gs64 container — token-exact vs CPU with resident dense (CUDA_DENSE=1) AND with routed experts in VRAM (CUDA_RELEASE_HOST=1, host copies freed), in both teacher-forcing and generation, 3/3 runs. Crucially the COLI_GPU_FAIL_AFTER=0 control proves 33 live GPU calls actually executed — the token-exactness is not a silent CPU fallback. That is the stricter test for the #298 failure mode. Verified locally: clean merge, clean build, token-exact unchanged (fp 32/32, int4 21/32).

… semantics

The merged JustVugg#298 gave the CUDA kernels per-group scale handling for fmt=4 —
without the same semantics a g64 container on Vulkan would decode with per-row
scales (the exact bug JustVugg#298 fixed). All three shaders gain a fmt==4 branch:
int4 nibble decode with one scale per gs inputs, applied to the packed-word
partial (host gates gs to multiples of 8 so a word never straddles a group;
per-row scaling is skipped like fmt=5). Group size flows as an explicit
parameter through the upload-triggering entries into ColiVkTensor and the
push constants; engine call sites pass QT.gs and the VK gates accept fmt=4
via VK_FMT_OK (word-aligned gs only — anything else stays on the CPU path,
which JustVugg#298 already fixed).

Harness: ref helpers generalized to runtime group size (g_ref_gs); fmt=4
cases at gs=64 across the real shapes (dense, o-proj, batch, expert_group,
matmul_pair, absorb incl. S=2 causal + window) plus a gs=32 sanity case.

The Metal GEMV shader (mm_gemv, backend_metal.mm) handled fmt 0-3 only.
Anything else -- including dev's grouped-int4 format (fmt=4: packed
int4 nibbles, same layout as fmt=2, but one f32 scale per `gs`-element
group along I instead of one scale per row; CPU reference
matmul_i4_grouped in quant.h, active CUDA tier in JustVugg#298/JustVugg#451) -- fell
through the shader's unconditional `else` arm and was decoded AS RAW
F32. This was a silent-wrong-answer defect, not just a missing-
acceleration gap: bind_gemv (used by the fused decode-attention
kernels, coli_metal_attn_decode/coli_metal_layer_decode) accepted a
per-weight fmt from its caller with no validity gate at all, so any
fmt=4 dense/attention tensor reaching that path would "succeed" (ok=1)
while silently computing garbage. coli_metal_matmul/coli_metal_gemm had
explicit fmt<=3 gates and so safely rejected fmt=4 (0/CPU-fallback);
bind_gemv had no such safety net.

COMMIT ORDERING (bisect-safety, see PR_BODY.md sec 1 for the full
argument): this commit lands FIRST, before the qt_from_disk allocator
fix (next commit). At this commit alone, the capability is CORRECT but
INERT: qt_from_disk's fmt=4 scale buffer is still allocated with a bare
falloc() (unregistered, not page-aligned), so resolve() in
backend_metal.mm can never find it, and every real fmt=4 dense/
attention tensor's scale buffer permanently misses bind_gemv/
coli_metal_gemm regardless of what this commit adds to the shader --
safe CPU fallback, not a wrong answer. No commit in this branch is ever
"the shader is fixed but a scale buffer resolves to the wrong/stale
allocation" -- fixed-but-unreachable, then reachable-because-fixed, in
that order, never the reverse. Verified empirically at this exact
commit (this branch, this session): both METAL=0/METAL=1 build clean,
0 warnings, and `make metal-test` passes 27/27 -- metal-test's fmt=4
cases construct their own tensors directly via posix_memalign +
coli_metal_register (never through qt_from_disk), so none of them
depend on the allocator fix landing first; the capability this commit
adds is exercised and proven correct here, just not yet reachable from
a real loaded model.

Shader: unlike fmt 1-3 (single `acc * scale[o]` after the full dot
product), fmt=4's group scale must be folded into the accumulation
per-group, before summing across groups -- this is matmul_i4_grouped's
actual math, not a simplification. New `constant int& gsz [[buffer(9)]]`
parameter carries the group size (ignored, harmlessly, for fmt!=4).
Each of the 32 SIMD lanes owns one packed byte (2 elements) per 64-wide
stride -- memory access stays coalesced and a group never splits a byte
(gs is always even, per the loader's candidate list). Deliberately not
vectorized like fmt=2's uchar4/float4 path -- see PR_BODY.md sec 7 for
why that tradeoff was made on purpose.

Host plumbing: gs threaded through every path that reaches mm_gemv --
coli_metal_matmul, coli_metal_gemm, and bind_gemv (-> AttnW ->
coli_metal_attn_decode / coli_metal_layer_decode) -- plus the colibri.c
call sites (matmul_qt_ex's Metal-GEMM gate now includes fmt==4;
attention_rows/layer_forward_rows pass .gs alongside each .fmt already
being passed, a no-op for every model not using fmt=4). fmt_bytes and
the new fmt_scale_bytes fix coli_metal_matmul's scale-buffer wrap size,
which was previously hardcoded to O floats (wrong for fmt=4's
O*ceil(I/gs)). kv_b (MLA absorption core) and the batched routed-expert
MoE path (moe_gemv/coli_metal_moe_block) are deliberately untouched --
both already gate fmt=4 to a safe CPU fallback today, and extending
them is a structurally separate kernel family; see PR_BODY.md sec 7 for
the full scope statement.

Tests (tests/test_backend_metal.mm): new cpu_ref_grouped (double-
precision oracle mirroring matmul_i4_grouped, same construction
tests/test_i4_grouped.c already established) and run_grouped harness,
using a magnitude-relative tolerance (justified in PR_BODY.md sec 5)
instead of the existing ymax-relative convention, since a grouped dot
product's cancellation can otherwise misreport as a kernel defect.
Covers the spec's shape matrix (I mult/non-mult of 64, I<=64
degenerate, S=1/S>1, outlier-heavy rows) via coli_metal_matmul, one
case via coli_metal_gemm (the large-batch path), and two via a new
run_attn_grouped harness that exercises the real fused-attention
bind_gemv path end-to-end (not just the standalone kernel entry point).
(These two attn cases have known blind spots as first committed here --
closed two commits later, see that commit's message and PR_BODY.md
sec 10; kept as originally written here for bisect fidelity.)

make metal-test: 15/15 (stock e9b3614) -> 27/27 (12 new fmt=4 cases;
original 15 unchanged -- the negative control). make glm METAL=0/
METAL=1: clean, 0 warnings, both before and after. make test-c /
test-python: unaffected, all pass. Full gates, evidence classes,
and DEVIATIONS/UNCERTAINTIES in PR_BODY.md (untracked, worktree root).

(cherry picked from commit f0fa5a9)
(cherry picked from commit 82984cdeef67ede71d48b32f2ddd4734f4622eba)

FIX ROUND 2, engine defect (clean-room conformance trial finding, the
headline item): qt_addrow and qt_matvec_rows (colibri.c) serve the kv_b
MLA-absorption CPU path. Both dispatch fmt 0/4/5 explicitly, then fall
through assuming a PER-ROW scale (t->s[row]) followed by fmt=1/2/3
(qt_addrow) or fmt=0/1/2/3/4/5 via an if/else-if chain ending in a bare
`else` (qt_matvec_rows) -- nothing stopped any OTHER fmt from reaching
that fall-through.

Reproduced (trial-verified, re-confirmed here): a fmt=7 QT (t->s holds
ceil(O/128)*ceil(I/128) per-block floats, not O -- a [130,130] tensor has
nblk=4, so t->s[row] overreads for every row past 3) reaches the
fall-through's `t->q4+(int64_t)row*((I+3)/4)` -- t->q4 is NULL for fmt=7
(raw bytes live in t->q8 instead, same convention as fmt=1) -- and
dereferences NULL-plus-offset: SIGSEGV. A fmt=6 QT (t->s is a FIXED
4-byte tag, qsalloc(1), not O floats) overreads t->s[row] for row>0 and
then silently misreads the real E8/IQ3 lattice bytes in t->q4 as
int2-packed data -- same bug SHAPE as JustVugg#298's CUDA absorb-kernel fix (this
file's own fmt=4/5 branches exist for exactly this reason; fmt=6 was
simply missed).

Both functions now refuse loudly (stderr naming the function and the
fmt, exit(1)) for any fmt they don't explicitly handle: qt_addrow gets an
explicit `fmt!=1 && fmt!=2 && fmt!=3` guard before the per-row-scale read
(fmt 0/4/5 already returned above); qt_matvec_rows' final bare `else`
becomes `else if(fmt==3)` plus a new refusing `else`. Reachability note
(context, not a scope excuse): both functions serve ONLY the kv_b absorb
path, and tools/repack_fp8_passthrough.py deliberately excludes kv_b_proj
from fmt=7 repacking -- so this fires only via a hand-slotted or
ambiguous-collision container, never this repo's own tooling's output.
Crash-instead-of-refuse is still a real defect (loud failure, every
refusal names its condition), and the fmt=7 QT surface these functions
can now be handed is one this same PR pair created.

tests/tests/test_qt_addrow.c (new): fork+pipe+waitpid refusal tests (this
suite's house pattern) for fmt=6 and fmt=7 through BOTH functions, using
the coordinator's own [130,130] partial-block repro shape for fmt=7; the
harness also detects a raw SIGSEGV (WIFSIGNALED) and reports it as the
specific failure it is, rather than hanging or mis-reporting. Byte-
identity checks for every format both functions still handle (0/1/2/3/4/5)
against an independently-written reference dequantizer (not copy-pasted
from either function -- restructured as a single per-element loop per
format), epsilon-compared (float multiplication reassociation, e.g.
qt_addrow's precomputed c=coef*scale vs the reference's coef*(scale*w[i]),
is not bit-exact even though mathematically identical -- confirmed by
inspecting actual failures at strict equality before relaxing, all
last-ULP-only, no shape/decode errors).

PROVEN TO BITE (RAN, full transcript in the report): reverted the guard in
qt_addrow only, rebuilt, ran the new test -- reproduced the exact SIGSEGV
(signal 11) for fmt=7 and the silent non-refusal for fmt=6, caught cleanly
by the test's own signal-aware harness. Reverted (diffed back to
byte-identical with the pre-mutation file), reran clean. Repeated
independently for qt_matvec_rows' guard alone (qt_addrow's guard left
intact) -- same SIGSEGV/silent-non-refusal pair reproduced and reverted.

RAN (this commit): make clean && make portable -> clean rebuild, zero
warnings. make test-c -> full C suite green, exit 0, includes the new
tests/test_qt_addrow binary. make test-python -> 153 tests, 10 skipped, 0
failures, exit 0. make glm METAL=1 -> clean rebuild, zero warnings. make
metal-test under COLI_METAL_RESSET=0 AND =1: both full suite green, exit
0. One-shot -Wno-unused-function-stripped build: same 8 pre-existing,
unrelated orphans as before this round, no new ones.

… semantics

The merged JustVugg#298 gave the CUDA kernels per-group scale handling for fmt=4 —
without the same semantics a g64 container on Vulkan would decode with per-row
scales (the exact bug JustVugg#298 fixed). All three shaders gain a fmt==4 branch:
int4 nibble decode with one scale per gs inputs, applied to the packed-word
partial (host gates gs to multiples of 8 so a word never straddles a group;
per-row scaling is skipped like fmt=5). Group size flows as an explicit
parameter through the upload-triggering entries into ColiVkTensor and the
push constants; engine call sites pass QT.gs and the VK gates accept fmt=4
via VK_FMT_OK (word-aligned gs only — anything else stays on the CPU path,
which JustVugg#298 already fixed).

Harness: ref helpers generalized to runtime group size (g_ref_gs); fmt=4
cases at gs=64 across the real shapes (dense, o-proj, batch, expert_group,
matmul_pair, absorb incl. S=2 causal + window) plus a gs=32 sanity case.

The Metal GEMV shader (mm_gemv, backend_metal.mm) handled fmt 0-3 only.
Anything else -- including dev's grouped-int4 format (fmt=4: packed
int4 nibbles, same layout as fmt=2, but one f32 scale per `gs`-element
group along I instead of one scale per row; CPU reference
matmul_i4_grouped in quant.h, active CUDA tier in JustVugg#298/JustVugg#451) -- fell
through the shader's unconditional `else` arm and was decoded AS RAW
F32. This was a silent-wrong-answer defect, not just a missing-
acceleration gap: bind_gemv (used by the fused decode-attention
kernels, coli_metal_attn_decode/coli_metal_layer_decode) accepted a
per-weight fmt from its caller with no validity gate at all, so any
fmt=4 dense/attention tensor reaching that path would "succeed" (ok=1)
while silently computing garbage. coli_metal_matmul/coli_metal_gemm had
explicit fmt<=3 gates and so safely rejected fmt=4 (0/CPU-fallback);
bind_gemv had no such safety net.

COMMIT ORDERING (bisect-safety, see PR_BODY.md sec 1 for the full
argument): this commit lands FIRST, before the qt_from_disk allocator
fix (next commit). At this commit alone, the capability is CORRECT but
INERT: qt_from_disk's fmt=4 scale buffer is still allocated with a bare
falloc() (unregistered, not page-aligned), so resolve() in
backend_metal.mm can never find it, and every real fmt=4 dense/
attention tensor's scale buffer permanently misses bind_gemv/
coli_metal_gemm regardless of what this commit adds to the shader --
safe CPU fallback, not a wrong answer. No commit in this branch is ever
"the shader is fixed but a scale buffer resolves to the wrong/stale
allocation" -- fixed-but-unreachable, then reachable-because-fixed, in
that order, never the reverse. Verified empirically at this exact
commit (this branch, this session): both METAL=0/METAL=1 build clean,
0 warnings, and `make metal-test` passes 27/27 -- metal-test's fmt=4
cases construct their own tensors directly via posix_memalign +
coli_metal_register (never through qt_from_disk), so none of them
depend on the allocator fix landing first; the capability this commit
adds is exercised and proven correct here, just not yet reachable from
a real loaded model.

Shader: unlike fmt 1-3 (single `acc * scale[o]` after the full dot
product), fmt=4's group scale must be folded into the accumulation
per-group, before summing across groups -- this is matmul_i4_grouped's
actual math, not a simplification. New `constant int& gsz [[buffer(9)]]`
parameter carries the group size (ignored, harmlessly, for fmt!=4).
Each of the 32 SIMD lanes owns one packed byte (2 elements) per 64-wide
stride -- memory access stays coalesced and a group never splits a byte
(gs is always even, per the loader's candidate list). Deliberately not
vectorized like fmt=2's uchar4/float4 path -- see PR_BODY.md sec 7 for
why that tradeoff was made on purpose.

Host plumbing: gs threaded through every path that reaches mm_gemv --
coli_metal_matmul, coli_metal_gemm, and bind_gemv (-> AttnW ->
coli_metal_attn_decode / coli_metal_layer_decode) -- plus the colibri.c
call sites (matmul_qt_ex's Metal-GEMM gate now includes fmt==4;
attention_rows/layer_forward_rows pass .gs alongside each .fmt already
being passed, a no-op for every model not using fmt=4). fmt_bytes and
the new fmt_scale_bytes fix coli_metal_matmul's scale-buffer wrap size,
which was previously hardcoded to O floats (wrong for fmt=4's
O*ceil(I/gs)). kv_b (MLA absorption core) and the batched routed-expert
MoE path (moe_gemv/coli_metal_moe_block) are deliberately untouched --
both already gate fmt=4 to a safe CPU fallback today, and extending
them is a structurally separate kernel family; see PR_BODY.md sec 7 for
the full scope statement.

Tests (tests/test_backend_metal.mm): new cpu_ref_grouped (double-
precision oracle mirroring matmul_i4_grouped, same construction
tests/test_i4_grouped.c already established) and run_grouped harness,
using a magnitude-relative tolerance (justified in PR_BODY.md sec 5)
instead of the existing ymax-relative convention, since a grouped dot
product's cancellation can otherwise misreport as a kernel defect.
Covers the spec's shape matrix (I mult/non-mult of 64, I<=64
degenerate, S=1/S>1, outlier-heavy rows) via coli_metal_matmul, one
case via coli_metal_gemm (the large-batch path), and two via a new
run_attn_grouped harness that exercises the real fused-attention
bind_gemv path end-to-end (not just the standalone kernel entry point).
(These two attn cases have known blind spots as first committed here --
closed two commits later, see that commit's message and PR_BODY.md
sec 10; kept as originally written here for bisect fidelity.)

make metal-test: 15/15 (stock e9b3614) -> 27/27 (12 new fmt=4 cases;
original 15 unchanged -- the negative control). make glm METAL=0/
METAL=1: clean, 0 warnings, both before and after. make test-c /
test-python: unaffected, all pass. Full gates, evidence classes,
and DEVIATIONS/UNCERTAINTIES in PR_BODY.md (untracked, worktree root).

(cherry picked from commit f0fa5a9)
(cherry picked from commit 82984cdeef67ede71d48b32f2ddd4734f4622eba)

FIX ROUND 2, engine defect (clean-room conformance trial finding, the
headline item): qt_addrow and qt_matvec_rows (colibri.c) serve the kv_b
MLA-absorption CPU path. Both dispatch fmt 0/4/5 explicitly, then fall
through assuming a PER-ROW scale (t->s[row]) followed by fmt=1/2/3
(qt_addrow) or fmt=0/1/2/3/4/5 via an if/else-if chain ending in a bare
`else` (qt_matvec_rows) -- nothing stopped any OTHER fmt from reaching
that fall-through.

Reproduced (trial-verified, re-confirmed here): a fmt=7 QT (t->s holds
ceil(O/128)*ceil(I/128) per-block floats, not O -- a [130,130] tensor has
nblk=4, so t->s[row] overreads for every row past 3) reaches the
fall-through's `t->q4+(int64_t)row*((I+3)/4)` -- t->q4 is NULL for fmt=7
(raw bytes live in t->q8 instead, same convention as fmt=1) -- and
dereferences NULL-plus-offset: SIGSEGV. A fmt=6 QT (t->s is a FIXED
4-byte tag, qsalloc(1), not O floats) overreads t->s[row] for row>0 and
then silently misreads the real E8/IQ3 lattice bytes in t->q4 as
int2-packed data -- same bug SHAPE as JustVugg#298's CUDA absorb-kernel fix (this
file's own fmt=4/5 branches exist for exactly this reason; fmt=6 was
simply missed).

Both functions now refuse loudly (stderr naming the function and the
fmt, exit(1)) for any fmt they don't explicitly handle: qt_addrow gets an
explicit `fmt!=1 && fmt!=2 && fmt!=3` guard before the per-row-scale read
(fmt 0/4/5 already returned above); qt_matvec_rows' final bare `else`
becomes `else if(fmt==3)` plus a new refusing `else`. Reachability note
(context, not a scope excuse): both functions serve ONLY the kv_b absorb
path, and tools/repack_fp8_passthrough.py deliberately excludes kv_b_proj
from fmt=7 repacking -- so this fires only via a hand-slotted or
ambiguous-collision container, never this repo's own tooling's output.
Crash-instead-of-refuse is still a real defect (loud failure, every
refusal names its condition), and the fmt=7 QT surface these functions
can now be handed is one this same PR pair created.

tests/tests/test_qt_addrow.c (new): fork+pipe+waitpid refusal tests (this
suite's house pattern) for fmt=6 and fmt=7 through BOTH functions, using
the coordinator's own [130,130] partial-block repro shape for fmt=7; the
harness also detects a raw SIGSEGV (WIFSIGNALED) and reports it as the
specific failure it is, rather than hanging or mis-reporting. Byte-
identity checks for every format both functions still handle (0/1/2/3/4/5)
against an independently-written reference dequantizer (not copy-pasted
from either function -- restructured as a single per-element loop per
format), epsilon-compared (float multiplication reassociation, e.g.
qt_addrow's precomputed c=coef*scale vs the reference's coef*(scale*w[i]),
is not bit-exact even though mathematically identical -- confirmed by
inspecting actual failures at strict equality before relaxing, all
last-ULP-only, no shape/decode errors).

PROVEN TO BITE (RAN, full transcript in the report): reverted the guard in
qt_addrow only, rebuilt, ran the new test -- reproduced the exact SIGSEGV
(signal 11) for fmt=7 and the silent non-refusal for fmt=6, caught cleanly
by the test's own signal-aware harness. Reverted (diffed back to
byte-identical with the pre-mutation file), reran clean. Repeated
independently for qt_matvec_rows' guard alone (qt_addrow's guard left
intact) -- same SIGSEGV/silent-non-refusal pair reproduced and reverted.

RAN (this commit): make clean && make portable -> clean rebuild, zero
warnings. make test-c -> full C suite green, exit 0, includes the new
tests/test_qt_addrow binary. make test-python -> 153 tests, 10 skipped, 0
failures, exit 0. make glm METAL=1 -> clean rebuild, zero warnings. make
metal-test under COLI_METAL_RESSET=0 AND =1: both full suite green, exit
0. One-shot -Wno-unused-function-stripped build: same 8 pre-existing,
unrelated orphans as before this round, no new ones.

Retarget PR JustVugg#377 onto current dev (origin/dev @ 4aca059) after JustVugg#391 split
glm.c into colibri.c + quant.h/sample.h/kv_persist.h/telemetry.h. Reconciles
four collisions:

- JustVugg#391 split: all glm.c hunks resited -- prof_*/g_prof_io/ProfBase live in
  colibri.c (NOT telemetry.h); tiers_emit/emap_emit -> telemetry.h (with a
  rammap_slot forward-decl); st_fd_is_tmpfs/st_fd_fs_magic -> st.h; .coli_kv
  state-dir -> serve_ctx_init/run_serve/run_serve_mux + main.
- DISK-CLASS (1f00142): prof_physical_read_bytes/ProfPhysicalWire merged ON
  TOP of dev's dc_* fields; PROF protocol line extended 9->17 fields
  additively; expert_load_impl 6-arg `demand` signature preserved; g_prof_io
  routed through prof_ssd_tensor_bytes (tmpfs-excluded).
- DUAL-SSD mirror (JustVugg#298/JustVugg#469): ESlot.backing hand-merged with dev's
  aslab/afslab; map_of_fd exact-length + MADV_HUGEPAGE composes with rep_bfd.
- int3 fmt=5 (JustVugg#168): rammap_bind_one reuses dev's qt_resolve_fmt/detect_group_size
  (inline fmt-detection dropped; duplicate detect_group_size not re-added).

Verified: make colibri 0 warnings; make check 187 tests pass (test_uring skips
in sandboxes via the PR's helpers; test_rammap builds against colibri.c and
passes). Default path byte-identical -- oracle-safe by construction.

Co-Authored-By: Claude <noreply@anthropic.com>

… semantics

The merged JustVugg#298 gave the CUDA kernels per-group scale handling for fmt=4 —
without the same semantics a g64 container on Vulkan would decode with per-row
scales (the exact bug JustVugg#298 fixed). All three shaders gain a fmt==4 branch:
int4 nibble decode with one scale per gs inputs, applied to the packed-word
partial (host gates gs to multiples of 8 so a word never straddles a group;
per-row scaling is skipped like fmt=5). Group size flows as an explicit
parameter through the upload-triggering entries into ColiVkTensor and the
push constants; engine call sites pass QT.gs and the VK gates accept fmt=4
via VK_FMT_OK (word-aligned gs only — anything else stays on the CPU path,
which JustVugg#298 already fixed).

Harness: ref helpers generalized to runtime group size (g_ref_gs); fmt=4
cases at gs=64 across the real shapes (dense, o-proj, batch, expert_group,
matmul_pair, absorb incl. S=2 causal + window) plus a gs=32 sanity case.