Can I use the Autotuning feature while enabling AutoTP in the configuration file?

I previously attempted to use both AutoTP and autotuning together. It appeared to work when I set "mp_size": 1 in the autotuning configuration. However, it failed when mp_size was set to match autotp_size. The error seemed related to a train batch size mismatch.

Even when it did run, the results were questionable. For instance, with the same train_micro_batch_size_per_gpu, I observed higher throughput when "autotp_size": 2" compared to no tensor parallelism at all, which seemed counterintuitive.

Additionally, it looks like the initial model profiling phase in autotuning uses ZeRO Stage 3 to estimate the minimum memory requirement. But AutoTP isn’t compatible with ZeRO Stage 3, right? I worked around this limitation by copying the profiling results from a configuration without TP, tricking autotuning into thinking the model had successfully completed profiling. However, this workaround isn’t ideal.