Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

glm: pin draft+verify to one kernel family during speculation; soften the MTP guard (#163)#294

Merged
JustVugg merged 1 commit into
JustVugg:devJustVugg/colibri:devfrom
bokiko:fix/spec-kernel-pinbokiko/colibri:fix/spec-kernel-pinCopy head branch name to clipboard
Jul 16, 2026
Merged

glm: pin draft+verify to one kernel family during speculation; soften the MTP guard (#163)#294
JustVugg merged 1 commit into
JustVugg:devJustVugg/colibri:devfrom
bokiko:fix/spec-kernel-pinbokiko/colibri:fix/spec-kernel-pinCopy head branch name to clipboard

Conversation

@bokiko

@bokiko bokiko commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What

Fixes the MTP acceptance regression in #163 by implementing the fix the thread converged on: draft and verify must compute the same function.

Three switches in glm.c were S-dependent, so the draft-side forwards (S=1) and the
verify batch (Sβ‰₯2) could silently run different kernel families:

  1. the int4 IDOT gate S>=g_i4s β€” asymmetric exactly on ISAs where g_i4s>1
    (x86: S=1 exact / Sβ‰₯2 IDOT; ARM SDOT and POWER are S-independent, which is why
    the failure signature reversed across ISAs);
  2. the S==1-only fused gate+up pair kernel;
  3. the Metal GEMM row threshold (S>=16, reachable by verify batches at DRAFTβ‰₯15).

SPEC_PIN=1 (default) pins every forward issued while model drafts are live to the
platform's S=1 kernel family, so draft and verify agree by construction β€” whatever
that family is on the host, per the thread's conclusion (a per-ISA "disable IDOT"
would fix x86 and re-break ARM). Prefill and non-speculative decode keep their
existing gates and their IDOT speed. SPEC_PIN=0 restores the old behavior for A/B.

Also softens the 24-proposal <10% guard (as suggested in the thread): instead of
latching g_draft=0 for the whole session, drafting pauses for 256 tokens and
re-arms on a fresh window β€” a future rounding regression degrades MTP instead of
silently shutting it down.

Validation

OP's exact config: "Summarize the plot of Romeo and Juliet in three sentences.",
--topp 0.7 --ngen 32, int8 MTP head, DRAFT=3, SEED=1234, single run per cell.

x86 β€” AMD Ryzen 9 9955HX (Zen 5, AVX-512 VNNI), 92 GB, Linux, --ram 78

arm SPEC_PIN=0 (old) SPEC_PIN=1 (this PR)
default kernels (IDOT on), greedy 8% (2/24) β†’ guard trip, 1.10 tok/fw 33% (16/48), 2.00 tok/fw
default kernels (IDOT on), sampled 8% (2/24) β†’ guard trip, 1.10 tok/fw 29% (15/51), 1.88 tok/fw
IDOT=0, greedy 18% (11/60), 1.60 tok/fw 18% (11/60), 1.60 tok/fw β€” output text byte-identical
IDOT=0, sampled 21% (12/57), 1.68 tok/fw 21% (12/57), 1.68 tok/fw β€” output text byte-identical

The pinned default restores the OP's pre-#68 numbers exactly: 33% (16/48),
2.00 tok/forward β€” the same cells he reported for main @ 1bdaeee. The greedy
off-distribution opening heals too: old default opens "### Plot Summary",
pinned default opens "Here's a concise summary of the plot of Romeo and Juliet…".
Where both forwards were already exact (IDOT=0), the pin is a proven no-op
(byte-identical text, identical accept counts).

ARM β€” Apple M4 Pro (NEON/SDOT), 48 GB, macOS CPU-only build, --ram 38

arm SPEC_PIN=0 (old) SPEC_PIN=1 (this PR)
default kernels (IDOT on), greedy 20% (12/60), 1.60 tok/fw 23% (13/57), 1.68 tok/fw
default kernels (IDOT on), sampled 18% (11/60), 1.60 tok/fw 18% (11/60), 1.60 tok/fw
IDOT=0, greedy 18% (11/60) 18% (11/60) β€” output text byte-identical
IDOT=0, sampled 21% (12/57) 21% (12/57) β€” output text byte-identical

On ARM the pin resolves to int4=idot int8=idot β€” today's healthy default family β€”
so nothing collapses and nothing regresses. The small greedy gain is principled, not
noise-mining: under the pin the rare S=1 main-layer forwards (iterations where no
draft is proposed) go through the same IDOT family as the verify batches instead of
the S==1-only fused-pair kernel, removing the last S-dependent switch.

Two cross-checks worth calling out:

  • The exact path is ISA-portable: the IDOT=0 cells are the same counts on both
    machines
    (11/60 greedy, 12/57 sampled at SEED=1234) β€” Zen 5 AVX-512 and M4 Pro
    NEON agree run-for-run at exact kernels, extending @ZacharyZcR's faithfulness
    trace across ISAs.
  • My earlier ARM IDOT=0 collapse (0/24) does not reproduce on current dev:
    four runs today (one seeded + three unseeded) give 21%, 27%, 27%, 24% β€” 0/24 is
    far outside that spread. That datapoint was n=1 on a build from before glm.c: 4-accumulator NEON SDOT for int8/int4 expert dots (2.4x/core on Apple Silicon)Β #264's
    4-accumulator SDOT rework merged; I'm not claiming causality, just updating the
    record: on today's dev both ARM families are healthy and S-independent.

(Note: the x86 rig had an idle ollama daemon resident during runs β€” no model loaded,
acceptance unaffected, tok/s at most slightly conservative.)

make test: 62/62 pass on both hosts. [SPEC_PIN] banner reports the pinned family
at first speculation, e.g. on Zen 5: int4=exact int8=idot; on M4 Pro: int4=idot int8=idot.

What this does NOT claim

  • CUDA path untouched: its dispatch condition is S-independent, hence already
    draft/verify-consistent.
  • Acceptance percentages under --topp remain non-portable across engine versions
    (discontinuous expert pruning) β€” the tables compare arms within one build only.
  • Speed: this is a correctness fix, not a speed win. On x86 the pinned default runs
    verify batches on exact int4 kernels (slower per forward) but needs ~45% fewer
    forwards (16 vs 29 per 32 tokens); measured end-to-end decode is neutral on this
    host (greedy 0.66β†’0.71 tok/s, sampled 0.67β†’0.65). What the fix buys is MTP that
    stays alive and correct β€” with IDOT prefill untouched.

Closes #163 (with the guard-softening suggestion from the thread included).

… MTP guard (JustVugg#163)

MTP acceptance collapses when the draft (S=1) and verify (S>=2) forwards do
not compute the same function. Three switches were S-dependent: the int4
IDOT gate (S>=g_i4s, asymmetric exactly on ISAs where g_i4s>1), the
S==1-only fused gate+up pair, and the Metal GEMM row threshold.

SPEC_PIN=1 (default) pins every forward issued while model drafts are live
to the platform S=1 kernel family, so draft and verify agree by
construction. Prefill and non-speculative decode keep their gates.
SPEC_PIN=0 restores the old behavior for A/B.

The 24-proposal <10% guard now pauses drafting for 256 tokens and re-arms
on a fresh window instead of latching off for the whole session.

Co-Authored-By: Claude <noreply@anthropic.com>
@JustVugg
JustVugg merged commit a6a9992 into JustVugg:dev Jul 16, 2026
@JustVugg

Copy link
Copy Markdown
Owner

Merged into dev. Verified locally: clean three-way merge against current dev + compiles with zero warnings (-Wall -Wextra). This is the right fix for the root of #163 β€” pinning draft (S=1) and verify (S>=2) to one kernel family while drafts are live makes the FP accumulation order agree by construction, and turning the permanent g_draft=0 latch into a windowed pause+re-arm means a transient dip no longer kills MTP for the session. Complements #293's COLI_CUDA_MTP (which handles the CUDA cold-expert path) without overlapping it. Thank you.

JustVugg added a commit that referenced this pull request Jul 16, 2026
@bokiko benchmarked the feature on three hosts and found every setting is
either no faster or no longer coherent. Reproduced here on a 25 GB box, and
the numbers are worse than "a quality/speed tradeoff":

  - budget=8 -> hellaswag 30% vs 90% with it off (25% is the chance floor)
  - budget=4 -> decode is literal noise ("The **1...: s2151:")
  - MTP acceptance 0%. Which experts survive the cap depends on cache
    residency at the moment of the forward, so draft and verify do not
    compute the same function -- the invariant #294 just established for
    #163, violated through cache state instead of kernel dispatch. SPEC_PIN
    cannot fix that and should not try: it is feature semantics, not
    dispatch.
  - 0.13 tok/s vs 0.30 baseline, while loading 14.66 experts per layer
    against a topk=8 baseline. The cap does MORE disk I/O than not using it.
    "~335 GB I/O saved" counts dropped experts, not bytes not read.

EXPERT_BUDGET>0 is now ignored unless EXPERT_BUDGET_EXPERIMENTAL=1, with the
measurements printed. The code stays compiled and developable: MoE-Spec
(arXiv 2602.16052) is not a wrong idea, this implementation just has no
point where it is both faster and correct. Re-enabling it by default needs a
quality number next to every speed number.

Also gates the cap to decode (S<=4), @woolcoxm's fix from #292/#298: during
prefill the batch union is 30-100+ experts, and capping to 4-8 drops most of
them, corrupting the hidden state and therefore the KV cache. Necessary but
not sufficient -- the run above already includes it.

Removes issue_budget.md, issue_diskio.md and issue_grouped_quant.md: design
notes in the repo root, unlinked from any README, whose only user-facing
content was "EXPERT_BUDGET=6-8 -- good speedup, minimal quality loss" and
"+83% decode at budget=4". Measurement says otherwise, and they were the
only thing on main telling anyone to switch this on. They stay in history.

Reported-by: bokiko <#303>
Co-Authored-By: woolcoxm <#292>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ZacharyZcR added a commit to ZacharyZcR/colibri that referenced this pull request Jul 16, 2026
…), EXPERT_BUDGET quarantine (JustVugg#303), PIN=auto (JustVugg#301), Ctrl-C soft stop (JustVugg#265), Windows wave
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.