Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit 3207040

Browse filesBrowse files
pytorchbotatalman
andauthored
[CD] Fix slim-wheel cuda_nvrtc import problem (#145614)
[CD] Fix slim-wheel cuda_nvrtc import problem (#145582) Similar fix as: #144816 Fixes: #145580 Found during testing of #138340 Please note both nvrtc and nvjitlink exist for cuda 11.8, 12.4 and 12.6 hence we can safely remove if statement. Preloading can apply to all supporting cuda versions. CUDA 11.8 path: ``` (.venv) root@b4ffe5c8ac8c:/pytorch/.ci/pytorch/smoke_test# ls /.venv/lib/python3.12/site-packages/torch/lib/../../nvidia/cuda_nvrtc/lib __init__.py __pycache__ libnvrtc-builtins.so.11.8 libnvrtc-builtins.so.12.4 libnvrtc.so.11.2 libnvrtc.so.12 (.venv) root@b4ffe5c8ac8c:/pytorch/.ci/pytorch/smoke_test# ls /.venv/lib/python3.12/site-packages/torch/lib/../../nvidia/nvjitlink/lib __init__.py __pycache__ libnvJitLink.so.12 ``` Test with rc 2.6 and CUDA 11.8: ``` python cudnn_test.py 2.6.0+cu118 ---------------------------------------------SDPA-Flash--------------------------------------------- ALL GOOD ---------------------------------------------SDPA-CuDNN--------------------------------------------- ALL GOOD ``` Thank you @nWEIdia for discovering this issue Pull Request resolved: #145582 Approved by: https://github.com/nWEIdia, https://github.com/eqy, https://github.com/kit1980, https://github.com/malfet Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> (cherry picked from commit 9752c7c) Co-authored-by: atalman <atalman@fb.com>
1 parent ca3c3a6 commit 3207040
Copy full SHA for 3207040

File tree

1 file changed

+7
-7
lines changed
Filter options

1 file changed

+7
-7
lines changed

‎torch/__init__.py

Copy file name to clipboardExpand all lines: torch/__init__.py
+7-7Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -316,21 +316,21 @@ def _load_global_deps() -> None:
316316

317317
try:
318318
ctypes.CDLL(global_deps_lib_path, mode=ctypes.RTLD_GLOBAL)
319-
# Workaround slim-wheel CUDA-12.4+ dependency bug in libcusparse by preloading nvjitlink
320-
# In those versions of cuda cusparse depends on nvjitlink, but does not have rpath when
319+
# Workaround slim-wheel CUDA dependency bugs in cusparse and cudnn by preloading nvjitlink
320+
# and nvrtc. In CUDA-12.4+ cusparse depends on nvjitlink, but does not have rpath when
321321
# shipped as wheel, which results in OS picking wrong/older version of nvjitlink library
322-
# if `LD_LIBRARY_PATH` is defined
323-
# See https://github.com/pytorch/pytorch/issues/138460
324-
if version.cuda not in ["12.4", "12.6"]: # type: ignore[name-defined]
325-
return
322+
# if `LD_LIBRARY_PATH` is defined, see https://github.com/pytorch/pytorch/issues/138460
323+
# Similar issue exist in cudnn that dynamically loads nvrtc, unaware of its relative path.
324+
# See https://github.com/pytorch/pytorch/issues/145580
326325
try:
327326
with open("/proc/self/maps") as f:
328327
_maps = f.read()
329328
# libtorch_global_deps.so always depends in cudart, check if its installed via wheel
330329
if "nvidia/cuda_runtime/lib/libcudart.so" not in _maps:
331330
return
332-
# If all abovementioned conditions are met, preload nvjitlink
331+
# If all abovementioned conditions are met, preload nvjitlink and nvrtc
333332
_preload_cuda_deps("nvjitlink", "libnvJitLink.so.*[0-9]")
333+
_preload_cuda_deps("cuda_nvrtc", "libnvrtc.so.*[0-9]")
334334
except Exception:
335335
pass
336336

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.