glm: pin draft+verify to one kernel family during speculation; soften the MTP guard (#163)#294
Merged
Merged
glm: pin draft+verify to one kernel family during speculation; soften the MTP guard (#163)#294
Conversation
β¦ MTP guard (JustVugg#163) MTP acceptance collapses when the draft (S=1) and verify (S>=2) forwards do not compute the same function. Three switches were S-dependent: the int4 IDOT gate (S>=g_i4s, asymmetric exactly on ISAs where g_i4s>1), the S==1-only fused gate+up pair, and the Metal GEMM row threshold. SPEC_PIN=1 (default) pins every forward issued while model drafts are live to the platform S=1 kernel family, so draft and verify agree by construction. Prefill and non-speculative decode keep their gates. SPEC_PIN=0 restores the old behavior for A/B. The 24-proposal <10% guard now pauses drafting for 256 tokens and re-arms on a fresh window instead of latching off for the whole session. Co-Authored-By: Claude <noreply@anthropic.com>
Owner
|
Merged into |
JustVugg
added a commit
that referenced
this pull request
Jul 16, 2026
JustVugg
added a commit
that referenced
this pull request
Jul 16, 2026
@bokiko benchmarked the feature on three hosts and found every setting is either no faster or no longer coherent. Reproduced here on a 25 GB box, and the numbers are worse than "a quality/speed tradeoff": - budget=8 -> hellaswag 30% vs 90% with it off (25% is the chance floor) - budget=4 -> decode is literal noise ("The **1...: s2151:") - MTP acceptance 0%. Which experts survive the cap depends on cache residency at the moment of the forward, so draft and verify do not compute the same function -- the invariant #294 just established for #163, violated through cache state instead of kernel dispatch. SPEC_PIN cannot fix that and should not try: it is feature semantics, not dispatch. - 0.13 tok/s vs 0.30 baseline, while loading 14.66 experts per layer against a topk=8 baseline. The cap does MORE disk I/O than not using it. "~335 GB I/O saved" counts dropped experts, not bytes not read. EXPERT_BUDGET>0 is now ignored unless EXPERT_BUDGET_EXPERIMENTAL=1, with the measurements printed. The code stays compiled and developable: MoE-Spec (arXiv 2602.16052) is not a wrong idea, this implementation just has no point where it is both faster and correct. Re-enabling it by default needs a quality number next to every speed number. Also gates the cap to decode (S<=4), @woolcoxm's fix from #292/#298: during prefill the batch union is 30-100+ experts, and capping to 4-8 drops most of them, corrupting the hidden state and therefore the KV cache. Necessary but not sufficient -- the run above already includes it. Removes issue_budget.md, issue_diskio.md and issue_grouped_quant.md: design notes in the repo root, unlinked from any README, whose only user-facing content was "EXPERT_BUDGET=6-8 -- good speedup, minimal quality loss" and "+83% decode at budget=4". Measurement says otherwise, and they were the only thing on main telling anyone to switch this on. They stay in history. Reported-by: bokiko <#303> Co-Authored-By: woolcoxm <#292> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ZacharyZcR
added a commit
to ZacharyZcR/colibri
that referenced
this pull request
Jul 16, 2026
β¦), EXPERT_BUDGET quarantine (JustVugg#303), PIN=auto (JustVugg#301), Ctrl-C soft stop (JustVugg#265), Windows wave
This was referenced Jul 16, 2026
This was referenced Jul 16, 2026
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the MTP acceptance regression in #163 by implementing the fix the thread converged on: draft and verify must compute the same function.
Three switches in
glm.cwere S-dependent, so the draft-side forwards (S=1) and theverify batch (Sβ₯2) could silently run different kernel families:
S>=g_i4sβ asymmetric exactly on ISAs whereg_i4s>1(x86: S=1 exact / Sβ₯2 IDOT; ARM SDOT and POWER are S-independent, which is why
the failure signature reversed across ISAs);
S>=16, reachable by verify batches at DRAFTβ₯15).SPEC_PIN=1(default) pins every forward issued while model drafts are live to theplatform's S=1 kernel family, so draft and verify agree by construction β whatever
that family is on the host, per the thread's conclusion (a per-ISA "disable IDOT"
would fix x86 and re-break ARM). Prefill and non-speculative decode keep their
existing gates and their IDOT speed.
SPEC_PIN=0restores the old behavior for A/B.Also softens the 24-proposal <10% guard (as suggested in the thread): instead of
latching
g_draft=0for the whole session, drafting pauses for 256 tokens andre-arms on a fresh window β a future rounding regression degrades MTP instead of
silently shutting it down.
Validation
OP's exact config: "Summarize the plot of Romeo and Juliet in three sentences.",
--topp 0.7 --ngen 32, int8 MTP head,DRAFT=3,SEED=1234, single run per cell.x86 β AMD Ryzen 9 9955HX (Zen 5, AVX-512 VNNI), 92 GB, Linux, --ram 78
The pinned default restores the OP's pre-#68 numbers exactly: 33% (16/48),
2.00 tok/forward β the same cells he reported for
main @ 1bdaeee. The greedyoff-distribution opening heals too: old default opens "### Plot Summary",
pinned default opens "Here's a concise summary of the plot of Romeo and Julietβ¦".
Where both forwards were already exact (IDOT=0), the pin is a proven no-op
(byte-identical text, identical accept counts).
ARM β Apple M4 Pro (NEON/SDOT), 48 GB, macOS CPU-only build, --ram 38
On ARM the pin resolves to
int4=idot int8=idotβ today's healthy default family βso nothing collapses and nothing regresses. The small greedy gain is principled, not
noise-mining: under the pin the rare S=1 main-layer forwards (iterations where no
draft is proposed) go through the same IDOT family as the verify batches instead of
the S==1-only fused-pair kernel, removing the last S-dependent switch.
Two cross-checks worth calling out:
machines (11/60 greedy, 12/57 sampled at SEED=1234) β Zen 5 AVX-512 and M4 Pro
NEON agree run-for-run at exact kernels, extending @ZacharyZcR's faithfulness
trace across ISAs.
four runs today (one seeded + three unseeded) give 21%, 27%, 27%, 24% β 0/24 is
far outside that spread. That datapoint was n=1 on a build from before glm.c: 4-accumulator NEON SDOT for int8/int4 expert dots (2.4x/core on Apple Silicon)Β #264's
4-accumulator SDOT rework merged; I'm not claiming causality, just updating the
record: on today's dev both ARM families are healthy and S-independent.
(Note: the x86 rig had an idle ollama daemon resident during runs β no model loaded,
acceptance unaffected, tok/s at most slightly conservative.)
make test: 62/62 pass on both hosts.[SPEC_PIN]banner reports the pinned familyat first speculation, e.g. on Zen 5:
int4=exact int8=idot; on M4 Pro:int4=idot int8=idot.What this does NOT claim
draft/verify-consistent.
--toppremain non-portable across engine versions(discontinuous expert pruning) β the tables compare arms within one build only.
verify batches on exact int4 kernels (slower per forward) but needs ~45% fewer
forwards (16 vs 29 per 32 tokens); measured end-to-end decode is neutral on this
host (greedy 0.66β0.71 tok/s, sampled 0.67β0.65). What the fix buys is MTP that
stays alive and correct β with IDOT prefill untouched.
Closes #163 (with the guard-softening suggestion from the thread included).