mistralai/Mistral-Small-4-119B-2603 is Mistral AI's latest open-source multimodal model featuring:

• Mixture of Experts (MoE): 128 experts, with 4 active per token, enabling efficient scaling and specialization.
• 119B total parameters, with 6B active parameters per token (8B including embedding and output layers).
• 256k context window, supporting long-form interactions and document analysis.
• Configurable reasoning effort: Toggle between fast, low-latency responses and deep, reasoning-intensive outputs.
• Native multimodality: Accepts both text and image inputs, unlocking use cases from document parsing to visual analysis.

Parallel Setup

We provide a fine-tuning recipe for Mistral Small 4 that scales training using Expert Parallelism and Pipeline Parallelism. The configuration runs with EP=8 and PP=4 across 4 nodes (8× H100 GPUs per node).

distributed:
  strategy: fsdp2
  tp_size: 1
  cp_size: 1
  pp_size: 4
  ep_size: 8

  sequence_parallel: false
  activation_checkpointing: false

  pipeline:
    pp_schedule: interleaved1f1b
    pp_microbatch_size: 1
    round_virtual_stages_to_pp_multiple: down
    scale_grads_in_schedule: false
    layers_per_stage: 4
    patch_inner_model: false
    patch_causal_lm_model: false

The base checkpoint is FP8-quantized on Hugging Face; NeMo AutoModel automatically dequantizes it during loading (dequantize_base_checkpoint: true).

Data

We use the MedPix-VQA dataset as an example. MedPix-VQA is a medical visual question answering dataset for training and evaluating VQA models in the clinical imaging domain, pairing radiological images from MedPix (a well-known medical image collection) with question–answer examples that target medical image interpretation.

Below is the loss curve obtained when fine-tuning on MedPix-VQA with this recipe: