@coderabbitai summary

Description

Fix a Python string truthiness bug in cpp/kernels/fmha_v2/setup.py that causes all SM89 E4M3 FMHA kernels to lose their precompiled cubins and fall back to source compilation, resulting in catastrophic accuracy regression on SM89 GPUs (L40S, L20, L40).

NVBugs: 5863806, 5868502, 5785465, 5785485
JIRA: TRTLLM-10833, TRTLLM-10867, TRTLLM-10257, TRTLLM-10258

Problem

FP8 accuracy tests fail catastrophically on SM89 GPUs:

Tests pass on H100 (sm90) but consistently fail on L40S, L20, and L40 (all sm89).

Git Bisect Result

Root Cause

The use_cubin_header() function in setup.py accepts a boolean enable_skip_softmax parameter, but callers pass a C++ string ('false') instead of a Python boolean:

'_skipSoftmax' in kname           → False        (Python bool)
pythonBoolean2cpp[False]          → 'false'      (C++ string literal)
use_cubin_header(..., 'false')                    (string passed as bool)
if 'false':                       → True          (non-empty string is truthy!)
    return False                                   (cubin disabled for ALL kernels)

This disables cubins for ALL SM89 E4M3 FMHA kernels (not just skip-softmax variants), forcing source compilation which has known accuracy regressions on SM89.

Fix

Separate the Python boolean from the C++ string: use enable_skip_softmax (Python True/False) for control flow, keep enable_skip_softmax_flag (C++ 'true'/'false') for code generation.

Impact

• SM89 E4M3 FMHA cubins are restored, fixing accuracy for all FP8 models on SM89 GPUs (L40S, L20, L40)
• SM90 cubins are restored for all existing kernel types. The only new kernel variant added since the bug (bidirectional sliding window, commit e699f232) lacks cubin entries, so use_cubin_header() now accepts an attention_mask_type parameter and skips cubins specifically for BIDIRECTIONAL_SLIDING_WINDOW kernels.
• Fixes NVBugs 5863806, 5868502 (accuracy component), 5785465, and 5785485

Test Coverage

• Direct verification: generated fmha_cubin.cpp correctly references cubin data instead of nullptr, 0 for SM89 E4M3 entries

PR Checklist

• PR description clearly explains what and why.

• PR Follows TRT-LLM CODING GUIDELINES.

• Test cases are provided for new code paths.

• Any new dependencies have been scanned for license and vulnerabilities.

• CODEOWNERS updated if ownership changes.

• Documentation updated as needed.

• Update tava architecture diagram if significant design change.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.