What does this PR do?

Fixes #3244 by releasing checkpoint-owned CUDA IPC state before colocated Megatron offload replaces model and optimizer storage.

With asynchronous NVRx checkpointing, the persistent checkpoint writer caches CUDA tensor handles when constant-structure caching is enabled. Finalizing the active save removes the request, but it does not release the writer-side cache. After offload replaces the model or optimizer storage, those handles keep the old GPU allocations alive. In the reported workload, this retained approximately 19.2 GiB per rank and caused the subsequent vLLM wakeup to fail with CUDA OOM.

This change:

• Finalizes pending checkpoint writes before both before-refit and after-refit offload paths replace CUDA storage.
• Limits persistent-writer teardown to the affected configuration: colocated generation, asynchronous NVRx, persistent checkpoint worker, constant-structure caching, and no CPU shared-memory checkpoint staging.
• Terminates the completed writer, clears FileSystemWriterAsync tensor caches, runs Python garbage collection, and releases CUDA IPC and allocator caches.
• Allows NVRx to recreate its persistent writer lazily for the next checkpoint.
• Adds lifecycle tests for both offload paths and for active/inactive cache cleanup behavior.

User impact

Asynchronous checkpointing remains enabled. The additional synchronization and cleanup only occur at storage-replacement boundaries for the affected colocated configuration. Other checkpoint strategies and non-colocated workloads retain their existing behavior.

Validation

• Targeted MCore tests: 4 passed, 15 deselected.
• Pre-commit checks: all applicable hooks passed, including Ruff, Ruff format, pyrefly, and recipe checks.
• Full VLM GRPO acceptance run: 4 nodes × 8 H100 GPUs × 20 training steps.
  • Step 10 and Step 20 asynchronous checkpoints both completed and finalized.
  • Post-checkpoint Megatron allocation decreased from approximately 27.04 GiB per rank in the failing run to approximately 7.85 GiB per rank.
  • The original post-checkpoint failure point passed, and training completed all 20 steps.
  • max(train/reward) = 0.8999998569488525 (> 0.75).
  • max(validation/accuracy) = 0.8343750238418579 (> 0.8).
  • Final scheduler status: COMPLETED, exit code 0:0.

Checklist

• Added regression coverage for the checkpoint/offload lifecycle.
• Ran targeted unit and full functional validation.
• No documentation update is required because this does not change a public API or configuration.