NanoVDB: fail with an actionable diagnostic when CUDA memory pools are unavailable by harrism · Pull Request #2256 · AcademySoftwareFoundation/openvdb

harrism · Jul 22, 2026

Addresses the failure mode in #2255 and complements #1798 / #1807's NANOVDB_USE_SYNC_CUDA_MALLOC. Related roadmap: #2232 (whose step 2 adds an injectable synchronous resource — the explicit, per-call form of the backend choice this PR keeps out of a hidden runtime path).

What

Fractional vGPU configurations (e.g. AWS g6f L4 instances) expose no stream-ordered memory pools — cudaDevAttrMemoryPoolsSupported == 0 — so NanoVDB's first cudaMallocAsync fails with cudaErrorNotSupported (801) with no indication of why or what to do.

Every device allocation in NanoVDB routes through the util::cuda::mallocAsync/freeAsync wrappers, so this PR improves the diagnostic at that single point without silently changing allocation semantics:

On a device without memory pools, mallocAsync emits an actionable message naming NANOVDB_USE_SYNC_CUDA_MALLOC and returns cudaErrorNotSupported — which every caller already surfaces through cudaCheck. freeAsync mirrors the capability check.
It does not silently fall back to cudaMalloc. A silent fallback would make an async-typed resource (DeviceResource, which advertises is_async_resource) quietly perform synchronous allocation — the type would misrepresent its runtime semantics, and a multi-stream caller would get unexpected serialization it never asked for. The choice of a synchronous backend belongs to the caller, explicitly: today via the macro, and (roadmap) via an injected synchronous resource under NanoVDB: injectable CUDA memory resources — design & roadmap #2232.
NANOVDB_USE_SYNC_CUDA_MALLOC remains the caller's compile-time opt-in to synchronous allocation, documented with the three-way selection and the one-definition-rule requirement (the wrappers are inline, so it must be defined identically in every TU). memoryPoolsSupported() exposes the cached per-device query.

Deploying on a pool-less device

Define NANOVDB_USE_SYNC_CUDA_MALLOC project-wide. A CI probe can automate this: compile-and-run a one-line cudaDevAttrMemoryPoolsSupported check on the target instance and set the flag when it reports 0 — capability detection performed by the deployer, not hidden in the library.

Testing

New unittest/TestUtilCuda.cu, built as two binaries — default (ctest nanovdb_cuda_util_unit_test) and -DNANOVDB_USE_SYNC_CUDA_MALLOC (ctest nanovdb_cuda_util_sync_unit_test) — covering both allocation modes. Tests: the cached query agrees with a direct per-device attribute query (and rejects out-of-range ids); and an allocate/use/free round-trip through the wrapper in whichever mode the build selects. The default binary's async round-trip skips on a device without memory pools (where those wrappers fail by design). Full suites green (CPU 153/153, nanovdb_test_cuda 53/53, memory-resource 8/8; compute-sanitizer clean; sm_89 / CUDA 12.6).

Non-CUDA builds

No impact — util/cuda/Util.h is CUDA-domain and unreachable from NANOVDB_USE_CUDA=OFF builds.

🤖 Generated with Claude Code

swahtz · Jul 23, 2026

I can confirm that this works on a g6f vGPU instance.

Fractional vGPU configurations commonly expose no stream-ordered memory pools (cudaDevAttrMemoryPoolsSupported == 0), so the first cudaMallocAsync fails with cudaErrorNotSupported. Rather than silently substituting synchronous cudaMalloc -- which would make an async resource misrepresent its own semantics, since the choice of a synchronous backend belongs to the caller -- the util::cuda::mallocAsync wrapper now detects the missing capability and emits an actionable diagnostic naming NANOVDB_USE_SYNC_CUDA_MALLOC before returning cudaErrorNotSupported, which every caller already surfaces via cudaCheck. freeAsync mirrors the check. The NANOVDB_USE_SYNC_CUDA_MALLOC macro remains the caller's explicit opt-in to synchronous allocation, and must be defined identically in every translation unit (the wrappers are inline). memoryPoolsSupported() exposes the cached per-device capability query, and mallocSync/freeSync name the synchronous pair the macro path maps to. TestUtilCuda.cu is compiled into two binaries -- default and macro-defined -- covering both modes; its async-path tests skip on a device without memory pools, where those wrappers fail by design. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Mark Harris <mharris@nvidia.com>

harrism · Jul 23, 2026

Thanks for validating on g6f — that confirms the detection and the synchronous path both work on the real hardware.

Heads up that I've since changed the behavior in this PR, so it's worth a re-test: instead of silently falling back to cudaMalloc on a pool-less device, the wrappers now fail with an actionable diagnostic and return cudaErrorNotSupported. The reasoning is that a silent fallback makes DeviceResource — which advertises itself as an async resource — quietly perform synchronous allocation, so the type would misrepresent its own semantics; the choice of a synchronous backend should be the caller's, explicitly.

So on g6f you'll now define NANOVDB_USE_SYNC_CUDA_MALLOC (which the error message names). To keep that zero-touch in CI, a configure-time probe can set it automatically — details on #2255. If you're able to re-run on g6f with the macro defined, that would confirm the deploy path end-to-end.

swahtz · Jul 24, 2026

I've rebuilt fvdb-core using the NanoVDB from this latest commit with NANOVDB_USE_SYNC_CUDA_MALLOC defined and can confirm that setting this flag works as expected on a g6f vGPU instance.

GridBuildThroughWrappers built a device grid through the wrappers, but PointsToGrid device builds are already covered by TestMemoryResource, TestNanoVDB, and TestMultiGPU. Its only unique coverage was the macro-forced-synchronous build, which is better established by the MallocAsyncRoundTrip test running in the NANOVDB_USE_SYNC_CUDA_MALLOC binary plus validation on pool-less hardware -- and it pulled the PointsToGrid builder header into a util-wrapper unit test. Remove it and the now-unused include. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Mark Harris <mharris@nvidia.com>

…lpers mallocSync/freeSync were the shared synchronous implementation for two callers: the macro tier and a runtime fallback. With the fallback removed in favor of failing loudly, only the macro tier remains, so the helpers are one-line pass-throughs with a single caller each. Inline cudaMalloc / cudaFree directly, restoring the macro-tier wrappers to their prior form and leaving memoryPoolsSupported as the only added function. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Mark Harris <mharris@nvidia.com>

swahtz · Jul 24, 2026

To test the expected behaviour without NANOVDB_USE_SYNC_CUDA_MALLOC defined, I recompiled and got the appropriate messge:

unit/test_accessors.py::TestAccessors::test_read_from_dense_cminor_0_cpu PASSED
unit/test_accessors.py::TestAccessors::test_read_from_dense_cminor_1_cuda NanoVDB: device 0 does not support stream-ordered CUDA memory pools required by cudaMallocAsync. Define NANOVDB_USE_SYNC_CUDA_MALLOC to allocate synchronously with cudaMalloc/cudaFree instead.
CUDA error 801: operation not supported (/home/ubuntu/fvdb-core/build/cp312-cp312-linux_x86_64-Release/_deps/nanovdb-src/nanovdb/nanovdb/cuda/DeviceResource.h:35)

My only note about the message should be to say "Define NANOVDB_USE_SYNC_CUDA_MALLOC when building …" to make it clear that this isn't a runtime environment variable that's respected.

swahtz

This looks like the right sized fix for this point in time for situations where a hardware configuration can't support the stream-ordered memory pool feature (time-share vGPUs) until we get the injectable memory allocator feature.

My only note is at the end changing the message to make it clear that the variable needs to be defined when building so the end-user doesn't assume we're referring to a runtime environment variable.

harrism requested a review from kmuseth as a code owner July 22, 2026 18:45

harrism mentioned this pull request Jul 22, 2026

Issue with DeviceResource (cudaMallocAsync) on vGPU without memory pool support #2255

Open

harrism force-pushed the nanovdb-vgpu-malloc-fallback branch from 957ca02 to e27f757 Compare July 22, 2026 20:12

harrism force-pushed the nanovdb-vgpu-malloc-fallback branch from e27f757 to 3178296 Compare July 23, 2026 19:55

harrism changed the title ~~NanoVDB: runtime fallback to synchronous allocation on devices without memory pools~~ NanoVDB: fail with an actionable diagnostic when CUDA memory pools are unavailable Jul 23, 2026

harrism mentioned this pull request Jul 23, 2026

NanoVDB: injectable CUDA memory resources — design & roadmap #2232

Open

5 tasks

swahtz added the nanovdb label Jul 23, 2026

harrism and others added 2 commits July 24, 2026 00:31

swahtz approved these changes Jul 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

NanoVDB: fail with an actionable diagnostic when CUDA memory pools are unavailable#2256

harrism commented Jul 22, 2026 •

edited

Loading

Uh oh!

swahtz commented Jul 23, 2026

Uh oh!

harrism commented Jul 23, 2026

Uh oh!

swahtz commented Jul 24, 2026

Uh oh!

swahtz commented Jul 24, 2026

Uh oh!

swahtz left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Search code, repositories, users, issues, pull requests...

Uh oh!

Conversation

harrism commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Deploying on a pool-less device

Testing

Non-CUDA builds

Uh oh!

swahtz commented Jul 23, 2026

Uh oh!

harrism commented Jul 23, 2026

Uh oh!

swahtz commented Jul 24, 2026

Uh oh!

swahtz commented Jul 24, 2026

Uh oh!

swahtz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

harrism commented Jul 22, 2026 •

edited

Loading