[nvidia-ctk-installer] do not revert cri-o config on shutdown #1360
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
29dca9c to
7d12529
Compare
This commit updates the behavior of the nvidia-ctk-installer for cri-o. On shutdown, we no longer delete the drop-in config file as long as none of the nvidia runtime handlers are set as the default runtime. This change was made to workaround an issue observed when uninstalling the gpu-operator -- management containers launched with the nvidia runtime handler would get stuck in the terminating state with the below error message: ``` failed to find runtime handler nvidia from runtime list map[crun:... runc:...], failed to "KillPodSandbox" for ... ``` There appears to be a race condition where the nvidia-ctk-installer removes the drop-in file and restarts cri-o. After the cri-o restart, if there are still pods / containers to terminate that were started with the nvidia runtime, then cri-o fails to terminate them. The behavior of cri-o, and its in-memory runtime handler cache, appears to differ from that of containerd as we have never encountered such an issue with containerd. This commit can be considered a stop-gap solution until more robust solution is developed. Signed-off-by: Christopher Desiniotis <cdesiniotis@nvidia.com>
7d12529 to
c9c7aa7
Compare
|
One question: Even if we don't delete the drop-in file, do we no still remove the binaries? |
elezar
reviewed
Oct 17, 2025
Comment on lines
+183
to
+185
| if !o.SetAsDefault { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Don't we still need to check whether we're using a drop-in file? We could be modifying the top-level config directly.
There was a problem hiding this comment.
I am not sure I follow why that matters? Regardless of whether we use a drop-in file or not, removing the nvidia runtime from the cri-o config can lead to the issues described in the PR description.
The reason I thought to add this conditional is that we must attempt to revert the config if nvidia is the default runtime, or else all future pods will fail to function (after the nvidia runtime binaries are removed).
I believe we still do, yes. |
tariq1890
reviewed
Oct 17, 2025
d32d3f9 to
e6d4b6e
Compare
…eanup Signed-off-by: Christopher Desiniotis <cdesiniotis@nvidia.com>
e6d4b6e to
521c243
Compare
tariq1890
reviewed
Oct 17, 2025
Signed-off-by: Christopher Desiniotis <cdesiniotis@nvidia.com>
tariq1890
approved these changes
Oct 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit updates the behavior of the nvidia-ctk-installer for cri-o.
On shutdown, we no longer delete the drop-in config file as long as
none of the nvidia runtime handlers are set as the default runtime.
This change was made to workaround an issue observed when uninstalling
the gpu-operator -- management containers launched with the nvidia
runtime handler would get stuck in the terminating state with the below
error message:
There appears to be a race condition where the nvidia-ctk-installer removes the drop-in file
and restarts cri-o. After the cri-o restart, if there are still pods / containers to terminate
that were started with the nvidia runtime, then cri-o fails to terminate them. The behavior
of cri-o, and its in-memory runtime handler cache, appears to differ from that of containerd as
we have never encountered such an issue with containerd.
This commit can be considered a stop-gap solution until more robust solution is developed.
Signed-off-by: Christopher Desiniotis cdesiniotis@nvidia.com