omp: reset CPU affinity before the tuning re-exec (fixes #471)#476
Merged
JustVugg merged 1 commit intoJul 21, 2026
JustVugg:devJustVugg/colibri:devfrom
veerareddyvishal144:fix-omp-reexec-affinityveerareddyvishal144/colibri:fix-omp-reexec-affinityCopy head branch name to clipboard
Merged
omp: reset CPU affinity before the tuning re-exec (fixes #471)#476JustVugg merged 1 commit intoJustVugg:devJustVugg/colibri:devfrom veerareddyvishal144:fix-omp-reexec-affinityveerareddyvishal144/colibri:fix-omp-reexec-affinityCopy head branch name to clipboard
JustVugg merged 1 commit into
JustVugg:devJustVugg/colibri:devfrom
veerareddyvishal144:fix-omp-reexec-affinityveerareddyvishal144/colibri:fix-omp-reexec-affinityCopy head branch name to clipboard
Conversation
execv preserves the CPU affinity mask. With user-exported OMP_PROC_BIND/OMP_PLACES, libgomp's constructor binds the master thread to place 0 before main() runs; the re-exec'd image inherited that one-core mask, enumerated OMP_PLACES=cores inside it, and jailed the whole team on one core (~20x slowdown, measured 0.11 vs 2.65 tok/s on a 2-socket 48C Ice Lake). Reset affinity to all online CPUs right before execv so the fresh libgomp binds from the full set; the user's OMP_* env still takes precedence. Verified on the same host: with OMP_PROC_BIND=spread OMP_PLACES=cores and no COLI_OMP_TUNED escape hatch, the re-exec'd engine now reports Cpus_allowed_list 0-95 and decodes at full speed. Adds the JustVugg#471 note to docs/tuning.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JustVugg
added a commit
that referenced
this pull request
Jul 21, 2026
#476's sched_setaffinity/CPU_* reset needs _GNU_SOURCE before the first system header. The standalone engine build defines it at the top of colibri.c (reset stays active where #471 needs it), but test TUs #include colibri.c after <assert.h>/<math.h>, so CPU_SETSIZE is exposed too late and the C test suite failed to compile (CPU_SETSIZE undeclared). Guard the block: active in make glm/colibri, compiled-out in tests (which never re-exec). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the fix you sketched in #471: reset the master thread's affinity to all online CPUs immediately before the
execv, so the second image's libgomp constructor enumerates places from the full core set and the user'sOMP_PROC_BIND/OMP_PLACESwork as documented. Clamped toCPU_SETSIZE, and asched_setaffinityfailure just warns and proceeds (the re-exec still happens, behaviour = status quo).Also adds the #471 note to
docs/tuning.mdnext to the "explicitOMP_*settings always take precedence" paragraph, withCOLI_OMP_TUNED=1documented as the escape hatch.Verified on the affected host (2× Xeon 8370C, 48C/2 NUMA, GLM-5.2 int4 fully resident)
Repro env from #471 (
OMP_NUM_THREADS=48 OMP_PROC_BIND=spread OMP_PLACES=cores, no escape hatch, re-exec taken —[OMP] hot-thread tuning: re-exec oncepresent in stderr):Cpus_allowed_list(engine, mid-decode)0-10-95Build: 0 warnings; oracle 32/32 TF passes.
One nuance from the issue thread, made explicit: a deliberate outer restriction (
taskset/numactl -Caround the whole process) is also widened by this reset. If that matters, gating the reset on "current mask ⊂ one core whileOMP_PROC_BINDis set" is a follow-up option; kept to your simpler version here.