## NVIDIA/Megatron-LM — v2.6…ykarnati-submodule-cc897f011

_7088+ commits._

### Features
- **[Main] Add the missing part to support 1F1B overlap for Qwen3-Next (#2997)** (4a23972)
- **Support EP with HSDP (#2840)** (473e283)
- **Add a unit test to check that RL `get_logprobs` will reuse training cudagraphed forward pass (#3209)** (18d69f1)
- **Add unit test for LatentMoE (#2892)** (7080697)
- **Add MimoOptimizer for heterogeneous parallelism** (1d817b6)
- **Add .worktrees/ to gitignore** (7da19e1)
- **Add unit tests for multi-module pipeline parallelism** (b46a157)
- **Add stage-aware forward pass to modality submodules** (997dfa5)
- **Add RankRole and ModuleStageInfo for multi-module pipeline parallelism** (7d566d9)
- **Revert "Add MTP support for hybrid models (#2363)"** (a0cc8ca)
- **Add MTP support for hybrid models (#2363)** (300d1b6)
- **Add a check to make sure we are distributing all the layers when using `--decoder-first-pipeline-num-layers` & `--decoder-last-pipeline-num-layers`  (#2947)** (4652e7b)
- **Revert "Multiturn rollout support prep (#2966)" (#3153)** (dbd8dda)
- **Multiturn rollout support prep (#2966)** (0b619c2)
- **add unit tests for multimodule pipeline schedules** (20d03f5)
- **support multimodule pipelining in 1f1b schedule** (0fa3dd8)
- **[training migration] Add StragglerDetectionConfig dataclass (#2435)** (008926a)
- **[training migration] Add CheckpointConfig dataclass (#2431)** (d4f9347)
- **Add end-to-end tests for M-FSDP and ND-Parallel (#3031)** (964c902)
- **add process group collection wrapper** (84ae4f0)
- **add pp stage checkers to p2p communicator** (c601de4)
- **Support NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 FP8/NVFP4 PTQ in example (#3079)** (b47c376)
- **add all_gather process-group for overlapping in fsdp disributed training (#2663)** (528cb2e)
- **Add router replay for MoE models (#2101)** (db6b895)
- **Reapply "Support multimodule communication (#2031)" (#3068)** (35e85a6)
- **[training migration] Add LoggerConfig dataclass (#2414)** (497d42d)
- **Add ability to save wgrads and dgrads (#3032)** (dbde759)
- **Revert "Support multimodule communication (#2031)" (#3068)** (0972f02)
- **Support multimodule communication (#2031)** (485ed18)
- **Add health endpoint to dynamic text gen server (#3009)** (93567e8)

### Fixes
- **Missing import fix (#3241)** (c036e77)
- **Fix coordinator address collision check in flask (#3208)** (9050d5b)
- **Fix missing PackedSeqParams import (#3214)** (bc2eb9a)
- **Fix MimoOptimizer to fail fast on missing modules** (a0acc15)
- **Fix TE 2.12 AllGather CI failure (#3101)** (1a61b77)
- **Fix seq pack flag in `get_logprobs` (#3206)** (94c9eae)
- **Fix linting issues: format code and remove unused imports** (0b6cefd)
- **Fix latent moe flops and backward_dw (#2977)** (a4008d0)
- **Fix for RL Test (#3148)** (c4bea0a)
- **Fix several syntax error (#3004)** (dff4189)
- **Fix bug in SFTDataset (#3185)** (50546da)
- **Revert "Fix two minor bugs in MTP implementation for hybrid models (#3194)"** (31d0c87)
- **Fix two minor bugs in MTP implementation for hybrid models (#3194)** (e836e62)
- **Fix H2D stream synchronization in optimizer offload (#3140)** (ae67076)
- **Fix missing argument in MoELayer.forward() (#3133)** (2fadde8)
- **cp: `Fix: nccl-ub in ddp path (3181)` into `main` (#3182)** (5d0a7fd)
- **[fix] Bug fix for offloading in evaluate() (#3043)** (a976754)
- **fix: Set --refit-method default to gloo (#3172)** (5415e1d)
- **[Community][Main] fix(moe): Fix theoretical memory calculation of layernorm. (#2434)** (9ad5906)
- **Fix SFTDataset truncation bug (#3158)** (7952d7e)
- **Fix for PR-2142 (#3165)** (4cd9563)
- **Fix coderabbit instructions error (#3150)** (f58b6d6)
- **Revert "Fix RL optimizer offload (#3112)" (#3141)** (3955c49)
- **Fix RL optimizer offload (#3112)** (287d2f4)
- **fix dim mapping in torch cat bridge comm** (b22f638)
- **Fix for PR-2142 (#3096)** (71c49b5)
- **[M-FSDP] Fix double buffering not working with activation recompute (#2689)** (b6b49e7)
- **Fix Multimodal Dockerfile (#3006)** (38cd9fc)
- **Fix GRPO re-fit functional test (#3113)** (f6c8a61)
- **Fix for Hybrid CP (#3091)** (fb6a592)
- **Fix bug of reuse_grad_buf_for_mxfp8_param_ag (#2802)** (33224cc)
- **fix(fsdp): add CLI argument for outer_dp_sharding_strategy (#3053)** (bb42a00)
- **fix ep weight gradnorm/num_zero calculation error for muon (#3024)** (b2e9390)
- **Fix broken functional tests in #2920 (#3071)** (369e0eb)
- **(REPLAY) Bug fix with --no-use-tokenizer-from-checkpoint-args (#3059)** (389436b)
- **Revert "Bug fix with --no-use-tokenizer-from-checkpoint-args (#3049)" (#3057)** (0683679)
- **Bug fix with --no-use-tokenizer-from-checkpoint-args (#3049)** (029f48f)

### Backend
- **[MIMO] Configure distributed optimizer process group in optimizer setup** (cc897f0)
- **Merge branch 'main' into yash/mimo-optimizer** (214e4fd)
- **Miscellaneous inference cleanup (Replay of !2955) (#3232)** (43db8c1)
- **enable cuda graph ut (#3197)** (982ca5d)
- **torch saver inference model offload (#3170)** (cd5ed74)
- **Do not let requests fail silently inside inference engine (#3228)** (e02344e)
- **Update copy-pr-bot.yaml [skip ci]** (4c48248)
- **Synchronize the request counts for EP inference with strict matching (#3033)** (1fdb29f)
- **Do not offload grad buffers when training graphs are enabled (#3231)** (27a5f83)
- **Simplify _get_pg_collection_for_optimizer to only fetch required groups** (6a5272e)
- **Update MimoModel for multi-module pipeline parallelism** (a1a8fdc)
- **Use global user buffer when the bucket size does not fit FixedPoolAllocator (#2857)** (afe443b)
- **mRoPE for MTP (#3114)** (70719cd)
- **Update README.md (#2111)** (1362e4a)
- **Handle `step` key correctly in checkpoint save with `--optimizer-cpu-offload` (#2874)** (f4502eb)
- **Revert "Miscellaneous inference cleanup (#2955)"** (0fe3232)
- **Miscellaneous inference cleanup (#2955)** (ffbc43f)
- **Update copy-pr-bot.yaml [skip ci]** (991c38f)
- **Update copy-pr-bot.yaml [skip ci]** (fdc04f6)
- **Nvshmem refit (#2696)** (14b70c7)
- **Vitalyk/multiturn v2 (#3167)** (b9ee19e)
- **Generate arguments from TransformerConfig (#2896)** (639c08a)
- **Automatically choose available ports in ZMQ (#2278)** (67f3515)
- **Update copy-pr-bot.yaml [skip ci]** (18deeff)
- **Force input ids generated by mock dataset are < vocab_size (#2945)** (063624b)
- **Merge branch 'main' into yash/1f1b_changes** (2d7c176)
- **Reapply 3955c49ed9af5e5b38dccdd30c1323c00b9bcd29 (#3146)** (36411dd)
- **Revise and move KD docs (#3108)** (4913c46)
- **Move kitchen extension file to private kitchen repository (#2779)** (190f5b6)
- **Create CodeRabbit config (#3131)** (9f05aac)
- **rename tensor conversion functions for clarity** (ebbb509)
- **rename module_collections to module_pgs for clarity** (b102eb7)
- **Disable Greptile status comments (#3127)** (d41bf66)
- **refactor multimodule pg collection and backward step** (a6606d8)
- **Refactor `rl_offload_kv_cache_during_training` to offload KV cache to CPU while retaining fixed virtual address  (#3048)** (42986ac)
- **handle 3d 2d tensor conversion in multimodule comm** (3badf57)
- **Update copy-pr-bot.yaml [skip ci]** (e2ff203)
- **remove retro (#3001)** (fc6969f)
- **Use the latest hybrid-ep (#3093)** (93ddc24)
- **Standardize RL unit tests (#3088)** (1453f94)
- **Minimize README contents (#3020)** (991138e)
- **Update copy-pr-bot.yaml [skip ci]** (6273d74)
- **Create greptile.json (#3087)** (4fb549f)
- **Move all bert and t5 tests to nightly (#3106)** (65217aa)
- **Logging cleanup (only log on rank 0 if possible) (#3036)** (6cf285b)
- **Update moe readme. (#2830)** (0888a06)
- **Inference functional tests: Write outputs to INFERENCE_OUTPUT_PATH instead of TENSORBOARD_PATH (#3061)** (4015ff1)
- **Harden GRPO functional tests (#3065)** (d68721b)
- **Reapply "Various CUDA graph improvements on capture time, replay time, memory footprint (#2572)" (#3056)** (642fdd9)
- **Added --ft-num-warmup-iters option. (#3052)** (06d0f46)
- **Optimizing post-processing of requests (#2920)** (53a2b19)
- **Keep FSDP's and DDP's finish_grad_sync API identical (#3070)** (3a7d74d)
- **Revert "Remove calculation of padding token in moe routing loss (#2142)" (#3069)** (4cfaa7d)
- **Update copy-pr-bot.yaml [skip ci]** (3593301)
- **Remove calculation of padding token in moe routing loss (#2142)** (10c6f01)
- **Update type hints and doc strings for moe_utils.py (#2821)** (03e0915)

### Docs
- **docs: improve Megatron-LM and Megatron Core descriptions (#3115)** (dceb1fb)
- **docs: Release docs (#3055)** (23a76d1)

### Chore
- **ci(hotfix): Pin uv (#3233)** (79e7bfe)
- **ci(fix): Parse unit tests in merge-queue (#3224)** (b477d12)
- **ci: Enable unit tests on merge-queue (#3186)** (0028273)
- **ci: Checkpoint retention (#3205)** (78475fe)
- **ci: Fix DSv3 (#3188)** (69a5c63)
- **ci: Add DSv3 proxy (#3169)** (a205538)
- **ci: Disable the api check for now (#3157)** (b168849)
- **ci(hotfix): Alert for GB200 (#3168)** (de15117)
- **ci: Onboard more GB200 tests (#3145)** (6de6362)
- **ci: Set NODE_RANK (#3143)** (f4af1bf)
- **ci: Add job timeouts (#3142)** (409af92)
- **build: Bump FLA (#3139)** (558fdaf)
- **build: Explicitly set minimum torch version to >= 2.6.0 (#3085)** (f0b1cb2)
- **ci: Add unit tests to merge queue (#3125)** (50132f2)
- **build: Use merge-commit-sha for container (#3123)** (a883e96)
- **ci: Mark test_compatible_with_nd_parallel as flaky (#3122)** (c22615e)
- **chore: rotate oncall schedule** (d5cac80)
- **build: Bump to TE2.12 (#3086)** (2b02a28)
- **ci: Add GPU health checks (#3100)** (2bdf7e1)
- **ci: Update improvement of step-time (#3104)** (dd83fc6)
- **ci: Repeat func tests, save logs of unit tests and lessen debug output (#3089)** (dea21a0)
- **ci: Disable gpt_dynamic_inference_tp1_pp1_dp8_583m_throughputtest_zmq (#3099)** (7031953)
- **ci: Log node name (#3081)** (94d8186)
- **ci: Mark test_mode_partial_cudagraph unit tests as flaky (#3064)** (55fe705)
- **ci: Skip test_precision_aware_optimizer (#3062)** (30dea5d)

_Recap by [Repo Wrapped](https://repowrapped.com/gh/NVIDIA/Megatron-LM?utm_source=github-action)._