Fix "context could not be synchronized" abort with CUDA + --threads#1538
Open
dsummer wants to merge 1 commit into
Netflix:masterNetflix/vmaf:masterfrom
codecmarket:fix/cuda-multithread-context-synccodecmarket/vmaf:fix/cuda-multithread-context-syncCopy head branch name to clipboard
Open
Fix "context could not be synchronized" abort with CUDA + --threads#1538dsummer wants to merge 1 commit intoNetflix:masterNetflix/vmaf:masterfrom codecmarket:fix/cuda-multithread-context-synccodecmarket/vmaf:fix/cuda-multithread-context-syncCopy head branch name to clipboard
dsummer wants to merge 1 commit into
Netflix:masterNetflix/vmaf:masterfrom
codecmarket:fix/cuda-multithread-context-synccodecmarket/vmaf:fix/cuda-multithread-context-syncCopy head branch name to clipboard
Conversation
Running `vmaf --gpumask 0 --threads N` aborts at flush with "context could not be synchronized" (exit 234), preceded by a flood of "feature ... cannot be overwritten at index N" warnings. The read dispatch in vmaf_read_pictures already notes that "multithreading for GPU does not yield performance benefits / disabled for now", but it is not actually disabled: when a thread pool is present the function returns early via threaded_read_pictures_batch, bypassing the HAVE_CUDA device- picture cleanup below it (the cuEventRecord(finished) and the unref of ref_device/dist_device), and feature extractors flush from worker threads that lack the bound CUDA context. Tear the thread pool down in vmaf_cuda_import_state so CUDA runs take the single-threaded path, matching the existing comment's intent. GPU feature threading yields no throughput benefit. Reproduced on master with an RTX 5090 / CUDA 13.2: before, exit 234; after, completes with scores identical to the single-threaded path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
|
Re-verified against the v3.2.0 release (
The branch is based on |
This was referenced Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Running
vmaf --gpumask 0 --threads Naborts at flush withcontext could not be synchronized(exit 234), preceded by a flood offeature ... cannot be overwritten at index Nwarnings.Cause
The read dispatch in
vmaf_read_picturescarries the note "multithreading forGPU does not yield performance benefits / disabled for now", but GPU feature
threading is not actually disabled. When a thread pool is present,
vmaf_read_picturesreturns early viathreaded_read_pictures_batch, which:HAVE_CUDAdevice-picture cleanup below it — thecuEventRecord(finished)and thevmaf_picture_unrefofref_device/dist_device; andcontext.
The CUDA context is therefore left unsynchronized and
flush_contextfails.Fix
Tear the thread pool down in
vmaf_cuda_import_stateso CUDA runs take thesingle-threaded path, matching the existing comment's intent. GPU feature
threading yields no throughput benefit (the GPU pipeline, not host
feature-extraction parallelism, is the bound).
Validation
RTX 5090, CUDA 13.2,
vmaf_v0.6.1, 1080p y4m:--gpumask 0(single-thread)--gpumask 0 --threads 4context could not be synchronized)Re-verified on the v3.2.0 release (master
ac9467ff)The bug still reproduces on current
master, and this fix still appliescleanly —
vmaf_cuda_import_stateis unchanged since this PR was opened, so thepatch cherry-picks without conflict. Built pure upstream
masterwith CUDA(
-Denable_cuda=true -Denable_float=true -Denable_nvcc=true) on an RTX 5090 /CUDA 13.3, 854p y4m:
--gpumask 0 --threads 085.194703)--gpumask 0 --threads 485.194703(bit-identical to single-thread)--threads 4(CPU, no CUDA import)