Summary

This PR turns the JetPack 7.2 image into the full accelerated Inference server and carries the tensor-native Workflow work previously reviewed in #2649.

• builds one production JetPack 7.2 / L4T R39.2 image for Orin (SM87) and Thor (SM110)
• adds hardware-aware GStreamer/RTSPS ingestion, FFmpeg, CUDA-enabled OpenCV 4.13, Jetson NVDEC/NVENC/nvJPEG integration, and CUDA tensor bridges
• builds PyTorch 2.13.0, TorchVision 0.28.0 with nvJPEG, ONNX Runtime 1.26.0 with CUDA/TensorRT, FlashAttention, and Triton 3.7.1 for both GPU architectures
• integrates Superiorvision 0.30.0.dev3 and Tracktors 2.6.0.dev2, including the real batched CUDA tracker executor
• keeps frames, detections, geometry, tracker state, and tracker IDs tensor-native through the hot path; CPU conversion is deferred to explicit serialization, annotation, or I/O boundaries
• keeps generic pip CUDA runtimes and build-only media toolchains out of the final image
• fixes the streaming InferencePipeline to reuse its existing Workflow executor for the engine lifetime

Why Superiorvision and Tracktors

The upstream Supervision and Trackers APIs remain the compatibility contract, but their numeric hot paths are NumPy/CPU-oriented. The forks preserve those public interfaces while changing the internal representation and execution model:

The compatibility aliases are deliberate: import supervision resolves to Superiorvision and import trackers resolves to Tracktors in this image. Runtime verification checks package identity, pinned versions, tensor-backed detections, CUDA execution, and the real CUDABatchExecutor/update_batch path.

Runtime contract

• JetPack / L4T: 7.2 / R39.2
• Python: 3.12
• CUDA architectures: SM87 and SM110
• CUDA, cuBLAS, cuSOLVER, cuDNN, TensorRT, and nvJPEG: supplied by the JetPack runtime
• GStreamer: RTSP, RTSPS, WebRTC, ICE, DTLS, SRTP, SCTP, plus available Jetson BSP codec elements
• media capabilities are detected at runtime; unavailable hardware encode/decode features use supported fallback paths instead of preventing startup
• only the full inference-server target is tagged for publication; the media target remains an internal build dependency

Persistent Workflow executor fix

The production camera/streaming path created a long-lived executor and exposed it to Workflow blocks, but did not pass it to ExecutionEngine.init(). The engine consequently created and destroyed worker pools for ready-step groups. CUDA streams are thread-local, so each replacement worker created new streams and left allocator blocks owned by streams that future workers could not reuse.

The observed signature was approximately 46–48 MiB of additional CUDA reservation per frame until a 5,284 MiB plateau. On Orin Nano this surfaced as roughly 5.75 GiB of cgroup kernel/NvMap memory, swap pressure, and OOM warnings even though live PyTorch allocations were small. A model-only reproduction showed the same growth, excluding Tracktors state as the cause.

Commit 86e76b04e passes the pipeline-owned executor to ExecutionEngine.init() and retains the existing shutdown owner. The regression test verifies that the same executor is supplied to both the engine and Workflow block initialization, remains alive during processing, and is shut down exactly once when the pipeline ends.

Allocator proof from the isolated reproduction:

• ephemeral workers: 20 → 71 → 119 → 168 MiB reserved over three frames, with about 6 MiB active
• persistent workers: 20 → 73 → 73 → 73 MiB reserved, with 8.6 MiB active
• torch.cuda.empty_cache() immediately released the inactive reservation

The HTTP Workflow path already supplied a shared executor; the affected path was InferencePipeline.init_with_workflow() used by streaming/camera pipelines.

Read-only Triton cache fix

Hardware validation found that the server could start under the documented read-only container configuration, but Tracktors' first fused Triton kernel compilation defaulted to /root/.triton and failed with OSError: read-only file system. Commit 5f9aaf60d defines TRITON_CACHE_DIR=/tmp/triton-cache in the JP7.2 runtime and makes the image verifier enforce that contract. The final device matrix used a read-only root filesystem and writable /tmp, injected no cache override, and confirmed that Triton generated the lifecycle, Kalman, and assignment PTX/CUBIN artifacts under the configured path.

Corrected Jetson benchmark

The comparison uses the same live RTSPS source, model, and Workflow. The NumPy path receives CPU/NumPy detections and uses the existing tracker implementation. The tensor path begins with data already resident on CUDA and uses Superiorvision + Tracktors, so the intended elimination of transfers and conversions is included rather than hidden.

• base image: roboflow/roboflow-inference-server-jetson-7.2.0@sha256:238c3f0a5dc0d409b3949d2cdebcfad864281ca5539d642e9a4f320bc10f372d
• corrected harness SHA-256: 8026b888d671bed5325d9c993395d5ddc3320ff52b44bd3f5d3d791b24f352fe
• model + tracker duration: 300 seconds per path per device
• model-only duration: 60 seconds per path per device
• primary metrics: delivered FPS and end-to-end wall latency
• resource telemetry: container/host CPU and memory, cgroup anon/file/kernel/swap, process RSS/PSS, GPU utilization, power, source/processed frames, and PyTorch active/reserved/max CUDA memory

Five-minute model + tracker results

Tensor path change versus NumPy

On Nano, the tensor path processed 6,345 of 6,356 source frames; NumPy processed 4,421 of 6,298 and dropped roughly 30%. GPU utilization per delivered frame improved by approximately 75% on Thor, 21% on AGX Orin, and 35% on Nano. The corrected tensor + tracker runs ended with about 34.3 MiB actively allocated by PyTorch; reservation peaked at 200 MiB and fell to 36 MiB after empty_cache(). There were no CUDA OOMs, allocator retries, or swap-pressure failures.

Validation completed

• current executor regression and full stream-pipeline module: 15 passed
• combined tensor correctness suite: 225 passed; SAM embed contract: 1 passed
• tracker Workflow suite: 21 passed, with one CUDA-only stream-handoff test skipped on the local non-CUDA runner
• seven reported stream/crop/keypoint/namedtuple/SAM/telemetry/grayscale regressions reproduced and covered by focused tests
• packed tracker IDs remain host-resident and aligned after unmatched-track filtering where host output is required
• submitter, scheduler, Tracktors, and consumer CUDA streams have explicit event and allocation-lifetime handoffs
• PyTorch wheel verification requires both sm_87 and sm_110
• runtime checks cover PyTorch, TorchVision/nvJPEG, ONNX Runtime CUDA/TensorRT, FlashAttention, Triton, OpenCV CUDA, GStreamer transports/codecs, Superiorvision, and Tracktors
• actionlint validates changed GitHub Actions workflows
• formatting, import ordering, and changed-file static checks pass
• corrected live tests completed on AGX Thor, AGX Orin, and Orin Nano
• final immutable digest starts under the documented read-only deployment on all three devices, remains healthy, and completes both shipped tensor/media verifiers without injected cache settings
• final device states: zero OOM kills, zero restarts, healthy after verification, and no fatal/traceback/crash log matches

Benchmark caveats

• The live stream was mostly a zero-detection workload. Active-track behavior, adversarial assignment, lifecycle, state, and ID parity are covered separately by the synthetic tracker correctness corpus.
• AGX Orin was source-capped at about 21 FPS, so equal throughput there means the tensor path delivered the full feed with substantially less CPU/GPU work; it is not the saturation limit.
• Workflow CUDA-event timings cross worker threads and streams and are diagnostic. FPS and end-to-end wall latency are the primary comparison.
• Continuous Docker/cgroup telemetry is the steady working-set measure. Endpoint RSS/PSS can expose LazyFree or CUDA-mapped pages and overstate live memory.
• The earlier benchmark that omitted the persistent engine executor is invalid and intentionally excluded from every table above.

Remaining release gates

• complete the ARM64 cloud rebuild from final PR head 5f9aaf60d: GitHub Actions run 29846434972
• publish roboflow/roboflow-inference-server-jetson-7.2.0:1.4.0-pr2651-torch213-5f9aaf60da
• record immutable index digest sha256:4b4700a5da766e58def0fa933dba10838e927f61050772bfb0403ed3322a10de and ARM64 manifest digest sha256:1248ae8e46fbe63c6f502d6bd41547668353aa41f7654071b6f4568b35994ae8
• run the read-only runtime smoke matrix against that exact digest on AGX Thor, AGX Orin, and Orin Nano
• confirm the runtime verifier rejects build-only toolchains, development payloads, duplicate pip CUDA runtimes, and missing pinned package/runtime capabilities
• record image size: 6.14 GB compressed registry payload (5.72 GiB), 16.8 GB unpacked on Jetson (about 15.6 GiB), 32 layers

Earlier builds remain useful as Depot cache inputs, but are not release candidates after the executor and read-only cache fixes. The immutable final digest above is the release candidate.