## pytorch/torchtitan — v0.1.0…v0.2.0

_273+ commits._

### Features
- **Add simpleFSDP numerics test to CI (#1911)** (1587868)
- **[TorchComms] Support training with EP (#1902)** (951f6ff)
- **[chore] add mi355x BF16 peak flops (#1909)** (e43621c)
- **[TorchComms] Support N-D parallelism device mesh setup (#1876)** (d15b59f)
- **add auto_eager_graph_pass (#1813)** (d0e2545)
- **add owners and CI status for experiments (#1859)** (304dfc3)
- **[vlm] Add light-weight CI for experimental models (#1848)** (06bbcd2)
- **add script to train with ft (#1812)** (42fe05f)
- **[VLM] Update config import from Llama 3 to support scaled RoPE args (#1849)** (abbb47a)
- **[VLM] Add token-imbalance loss (#1803)** (98d904f)
- **Add support for AC budget API (#1731)** (b276387)
- **[Llama] Add scaled RoPE support for Llama 3 and 4 (#1839)** (5bbda42)
- **[Qwen3] StateDictAdapter support for MoE model (#1766)** (4409c13)
- **add support for simplefsdp+ep (#1529)** (37e536d)
- **[mxfp8 moe training] add MX MoE model converter using torchao mxfp8 moe training; refactor quantization APIs to use unified API (#1701)** (5b5d468)
- **[Flux] Remove generate_image script and add CI test for inference  (#1746)** (7e805a9)
- **[Qwen3] Add 32b training configs (#1690)** (d66b72a)
- **[RFC] Support full bf16 training (#1646)** (e99e16c)
- **[Qwen3] Qwen3 MoE initial support  (#1685)** (bd3850b)
- **[Qwen3] Add 1.7b Config (#1671)** (78d4314)
- **[Refactor] Modular Integration Test Framework with DeepSeek-v3 Support (#1431)** (accc702)
- **Add Forge Code Owner (#1665)** (4e308c0)
- **Add config to AC to toggle early-stop and revert A2A autograd.Function workaround (#1580)** (25413d2)
- **Add CODEOWNERS for experiments/forge (#1655)** (55e42c0)
- **add train file specification option to run_start.sh** (bb1e576)
- **add wandb team entity and run name options (#1643)** (17ef753)
- **add model_parts ref to MetricsProcessor (#1578)** (7d744b2)
- **[refactor] support compile model and loss separately (#1608)** (08b8b24)
- **[HF] Model Definition Conversion Support for FLUX (#1582)** (c0b2e5a)
- **[Qwen3] Switch to verified RoPE implementation + Add weight tying support (#1590)** (7f1fa48)
- **[DeepSeek] add torch.compile + async TP (#1588)** (9e24689)
- **Add DualPipeV (#1571)** (803906b)
- **[EP] add support for ETP=1 (#1555)** (aeb3a4b)

### Fixes
- **Fix HF -> Torchtitan Expert Conversion Sorting Bug (#1918)** (fb54971)
- **Deepseek-V3 toml file minor fix (#1894)** (5fb7cc2)
- **[CP] Fix incorrect indentation (#1880)** (92ed8b3)
- **fix MoE TP backward (#1878)** (3ff7551)
- **Fix num of layers for deepseek-v3 (#1845)** (b6ccf22)
- **Fix docstring formatting (#1843)** (44e9218)
- **[simplefsdp] fix simplefsdp gradient_divide_factor (#1793)** (f014f31)
- **Fix async_with_pinned_mem doesn't set staging correctly (#1783)** (8ec37d2)
- **[BE] Fix minor typos in the release document (#1780)** (fa21894)
- **Fix CI link in README (#1779)** (76a9293)
- **Fix forge float8 import (#1770)** (a310420)
- **fix: validation loss accumulation rescaling (#1758)** (24e9105)
- **fix Qwen3 30B-A3B model configs (#1760)** (5359d98)
- **fix: pp grad accumulation is broken (#1732)** (3e1b843)
- **followup fix to #1740 (#1747)** (8d20f02)
- **fix: datasets broken import due to HF package and folder name collision (#1730)** (686b523)
- **Fix flux image_generation_test.  Change input dim to match encoder (#1726)** (22d2d44)
- **Fix EP token group padding issue (#1718)** (60645bc)
- **[Qwen3] Fix model arg computation for MoE model (#1704)** (d240be0)
- **fix: moe load balancing pp edge cases (#1689)** (6228f22)
- **fix broken tests (#1669)** (07c00d9)
- **fix small typo in type hint (#1667)** (fe621cb)
- **Fix toml files (#1659)** (4256f75)
- **Fix variable name in NotImplementedError message (#1637)** (ad06609)
- **[Qwen3] Fix weight tying for Qwen3 according to Huggingface configs (#1633)** (030879f)
- **fix(dataloader): Prevent RuntimeError from DataloaderStopIteration (#1627)** (9197908)
- **async tp minor fix (#1629)** (2025abb)
- **fix qwen3 compile config in parallelize.py (#1623)** (255a6ab)
- **Fix Typo (#1611)** (fd23080)
- **[EP] bug fixes (#1586)** (9233d83)
- **quick fix dsv3 fsdp (#1575)** (6fc499f)

### Backend
- **Bump version to 0.2.0 (#1919)** (29624e3)
- **[TorchComms] Update readme (#1917)** (b206439)
- **[TorchComms] Update readme (#1913)** (7209894)
- **deprecate input_batch in model inputs (#1896)** (db82f8b)
- **[TorchComms] update readme (#1877)** (6bccdb6)
- **Remove self.topk in ReordererSequenceParallel to prevent mutation (#1873)** (4faff04)
- **[refactor] graduate custom_config_module and unify args/config naming (#1871)** (7c10480)
- **Remove unnecessary all-gather qwen3 (#1870)** (8c1d1c5)
- **Remove torchao nightly version check in mx component (#1872)** (58c5455)
- **consolidate experiments/deepseek_v3 (#1869)** (8631e19)
- **move PP API to model agnostic file (#1868)** (6e06a49)
- **consolidate VLM data loading (#1866)** (248aca2)
- **graduate llama4 to core (#1865)** (eca011c)
- **TorchTitan e2e test on torchcomms device mesh (#1847)** (cd304c7)
- **Graduate qwen3 from experiment to core (#1860)** (d880de2)
- **allow disabling ft checkpoints (#1810)** (a82b77a)
- **minor refactor over EP (#1854)** (9603872)
- **Minor fixes in simple_fsdp experiments (#1853)** (dfd0a59)
- **Refactor attention and make attention mask an argument to the model (#1776)** (6a3a9da)
- **refactor TrainSpec to remove the name field (#1850)** (aa000a3)
- **enhance profiler config (#1809)** (21739fd)
- **Disable FlexAttention max-autotune when deterministic is used (#1808)** (41eff53)
- **[DSV3] Offload dequantization process to DCP QuantizedHFReader (#1804)** (a6f0cfc)
- **Provide load_seed_checkpoint_only option (#1800)** (eb13ba2)
- **Remove create_cp_block_mask (#1798)** (96149f6)
- **Update README.md (#1796)** (5217163)
- **Set proper deepseek-v3 16b parameters (#1792)** (82a9530)
- **Fake balanced routing in MoE (#1670)** (99fee81)
- **[Forge] Update ForgeJobConfig to use proper JobConfig entries (#1788)** (0cfbd0b)
- **Downgrade python version for pyproject.toml (#1787)** (d7169c9)
- **config compile backend (#1768)** (177b050)
- **bump up python version to 3.12 (#1778)** (5d8e2d5)
- **[Refactor] Refactor model nparams calculation into single static function (#1767)** (c854924)
- **port true bf16 training into forge experiment (#1775)** (9f3fe08)
- **[quantization] Refactor quantization API (#1763)** (8a2686c)
- **[CI] Disable Async TP CI (#1756)** (82f0287)
- **Separate SAC Wrapping of MoE and Attention Modules to Enable Flex Attention Compilation (#1683)** (0943771)
- **Allows to configure flight recorder file prefix (#1748)** (0f34257)
- **[RFC] Require bitwise equivalence for SimpleFSDP numerics (#1743)** (78841bf)
- **Implicit overlap of shared expert compute and token combine communication (#1741)** (476a965)
- **[refactor] dynamically import TrainSpec (#1740)** (5a8256c)
- **removes duplicate line (#1739)** (85d92de)
- **Grammatical updates to GC logging (#1737)** (5ec8765)
- **VLM: Onboarding native resolution, native aspect ratio, interleaved VLM training (#1615)** (c9cb304)
- **Removing call to init_weights in model's __init__ (#1711)** (40a8725)
- **Enable Simultanous WANDB and Tensorboard logging in torchtitan** (612e4a1)
- **small nit fixes (#1695)** (71dea16)
- **logging: clear root handlers to prevent duplicate logs and use stdout for StreamHandler (#1686)** (99c0cb2)
- **Save _to_copy and a2a in selective AC policy (#1672)** (c4e2291)
- **[BE] Move NoParallel to torchtitan.distributed (#1641)** (ea4989e)
- **Make SAC save_list configurable (#1675)** (2179939)
- **[DSV3] GroupedExperts weights conversion optimization (#1639)** (52a4c17)
- **[HF] Llama4 Text State Dict Adapter (#1662)** (298bf48)
- **seperate training termination from training loop** (3b9df60)
- **Enable multi rank safetensor consolidation (#1625)** (d484458)
- **[moe][compile] Turn capture_scalar_outputs off by default (#1649)** (88dfcbe)
- **Activation Checkpoint improvment (#1645)** (8a6c9fe)
- **enforce existence of .toml args in JobConfig dataclasses (#1644)** (eadca48)
- **update warning message (#1648)** (7156416)
- **Solving the validation hanging issue (#1634)** (a481c26)
- **Adding StateDictAdapter  (#1601)** (e65ef30)
- **Update torchft.md (#1596)** (4191def)
- **[Cleanup] Miscellaneous Refactors (#1607)** (cd337db)
- **Centralize Async TP Enablement with maybe_enable_async_tp API (#1619)** (cab22e7)
- **Switch DeepSeekV3 to Use FlexAttention by Default (#1610)** (f738a03)
- **Move the call to init_attention_mask to trainer (#1616)** (8a749c6)
- **improve MoE bias update logic in optimizer (#1593)** (2bfcdd8)
- **[DSV3] Upgrade to DeepSeek-V3.1 (#1609)** (82d6c3b)
- **Deprecate Llama Conversion Script (#1603)** (46a32e7)
- **Change freq_cis from persistent buffer to non-persistent buffer (#1600)** (9874e84)
- **[doc] update DeepSeekV3ModelArgs doc string (#1598)** (084d307)
- **[HF] Deprecate `tokenizer_path` in Toml Files (#1592)** (b5b7ffb)
- **[dsv3] Remove dtype to avoid confusion (#1599)** (9f47ceb)
- **MoE explicit prefetching in FSDP (#1594)** (a54725c)
- **added example for bidirectional checkpoint testing (#1540)** (e4847c8)
- **Adding Qwen3 model to the experiments folder (#1429)** (f9e8897)
- **[EP] remove token split overhead from DTensor in TokenReorderer pre hook (#1587)** (0d1b80d)
- **Added doc for Val/Eval and lm_eval integration (#1573)** (72b16b1)
- **added better guidance for if deprecated tokenizer path fails (#1568)** (a59abea)
- **Ignore tokenizer_path if it is an empty string (#1579)** (297a72a)
- **Use PYTORCH_ALLOC_CONF as PYTORCH_CUDA_ALLOC_CONF is deprecated (#1577)** (e629fe5)
- **[MoE/EP] apply dim-1 FSDP sharding for routed experts and rewrite shared experts with FFN (#1561)** (7354848)
- **llama4: Avoid staticmethod nested graph break for MoE compile (#1565)** (6377dce)

### Docs
- **docs: fix typos in documentation and code comments (#1898)** (f7a56e6)

### Chore
- **ci: add codespell pre-commit hook (#1899)** (4db8f6e)
- **refactor: use a common DatasetConfig class (#1712)** (be2c83d)

_Recap by [Repo Wrapped](https://repowrapped.com/gh/pytorch/torchtitan?utm_source=github-action)._