What's Changed

• FSDP2: Integration with LR schedulers, activation checkpointing, MFU support, and tests — @le1nux, @flxst, @fromm-m, @mali-git
  #316, #317, #319, #320, #343, #345, #346, #347, #350, #351, #355, #356, #359, #360, #377

• Parallelism: Added tensor parallelism and improved configuration handling — @le1nux
  #374, #412

• Training & Preprocessing: Scalable preprocessing, seeded shuffling, benchmarking, and profiling tools — @le1nux, @mali-git
  #290, #291, #295, #389, #360

• Model Features: Configurable SwiGLU dimensions, RoPE base frequency, weight tying, and checkpoint loading improvements — @mali-git, @flxst
  #289, #297, #321, #318, #305

• Instruction Tuning: 2025 update and communication test before training — @lllAlexanderlll, @rrutmann
  #379, #386, #385

• Bug Fixes: Tokenization consistency, Hugging Face conversion, SwiGLU shape handling, MFU accuracy, and GitHub tests — @BlueCrescent, @CYHSM, @flxst , @le1nux
  #280, #281, #282, #283, #288, #322, #394, #396, #390, #410, #382, #365

• Infrastructure: Updated Torch/Python versions, improved automation, and CI workflows — @le1nux, @flxst
  #273, #299, #413, #362, #363, #323, #302, #301, #303

New Contributors

• @CYHSM made their first contribution in #396

Full Changelog: v0.3.2...v0.4.0