## pytorch/torchtitan — v0.2.0…v0.2.1

_138 commits._

### Features
- **add safety checks for varlen  (#2179)** (5077be6)
- **Support TP when using vLLM engine to run inference w/ torchtitan model definition (#2165)** (a95d203)
- **[RL] Support Trainer and Generator Unified Model (#2174)** (30ab580)
- **Add rocm support for models, flux & torchft integration tests. (#2172)** (a452121)
- **Add Dependabot for GitHub Actions updates (#2163)** (b21555f)
- **Use new DeviceMesh unflatten to rewrite parallel_dims (#1660)** (183a0d2)
- **Add local built pytorch path for pyrefly (#2155)** (c08fa57)
- **[GPT-OSS] Add HF state dict adapter to support loading from HF checkpoints (#2021)** (64997d2)
- **[MoE] Add node limited routing support (#2111)** (2aac20a)
- **[Autoparallel] Add local_map variant of DSv3 and 2D mesh AP (#2129)** (995154f)
- **Support rope cache indexing using positions (#2112)** (f1d41a1)
- **Add CI for Autoparallel experiment llama3 on 4 GPUs (#2105)** (1ebd914)
- **Add warning to run_tests (#2123)** (b41832a)
- **[mxfp8] [docs] [BE] add MXFP8 usage documentation and benchmarks (#2096)** (b3da1a2)
- **add all SDPA backends to op_sac_save_list (#2095)** (b39377f)
- **[FLUX] Add FLUX inference test in CI (#1969)** (cbdb311)
- **add varlen attention for qwen 3 (#2084)** (1b9cfda)
- **Enhance loss_compare.py: Add Import/Export Options and Enable CI Comparison with Existing Losses (#2063)** (c70310c)
- **CUDAGraph support for SimpleFSDP and TP (#2050)** (f5e3a84)
- **[compiler toolkit] Add Trainer subclass for compiler toolkit (#2064)** (f541d91)
- **Add a loss comparison script (#2029)** (c8ebd7a)
- **Add export_dtype parameter to `convert_to_hf` function (#2041)** (22e959a)
- **[SimpleFSDP] add manual bucketing pass (#1881)** (d9bdfbb)
- **Add .claude to .gitignore (#2026)** (4b2b31c)
- **[compiler toolkit] Add tests and scripts for numerics check (#2015)** (cbfb8e1)
- **Add dry run mode (#2012)** (f5d2b18)
- **[Full DTensor][Reland] Add full_dtensor flag (#2013)** (e37f83f)
- **[SimpleFSDP] Add typing to simple_fsdp.py (#2001)** (fddd9eb)
- **[TorchComms] add testing badge at experiments readme (#2010)** (ae03494)
- **Add post_dataloading_processing method to Trainer (#1985)** (157d30d)
- **Add deterministic RL training experiment with vLLM (#1975)** (35bffe9)
- **[compiler] Switch to the new dynamo export API. (#1981)** (268020d)
- **add blurb for AMD's fork (#1997)** (ad78ed8)
- **[simplefsdp] add reshard after forward option (#1961)** (29eb910)
- **Enable ROCm CI support (#1786)** (7fcfdf5)
- **[compiler toolkit] Add integration tests for compiler toolkit experiments (#1944)** (dc0ccb2)
- **add .env to gitignore (#1943)** (444b58b)
- **[Reland][Compiler Toolkit] Add annotations to MoE (#1937) (#1941)** (efecf0c)
- **[TorchComms] add compile in ci test (#1930)** (7929410)
- **Add Context Parallelism to Flux model training (#1851)** (75d4e4d)

### Fixes
- **Fix qwen3 attention scaling calculation (#2173)** (29aafb9)
- **Fix pypa/gh-action-pypi-publish version to use SHA pinning (#2161)** (4438764)
- **Fix apply_compile called multiple times in PP initialization (#2135)** (7a398ea)
- **fix `ForgeEngine` compatibility issue with (#2121)** (303f284)
- **fix mxfp8 loss image (#2104)** (341b155)
- **compiler_toolkit: fix args access (#2067)** (d167a20)
- **Fix integration test gpu_arch_type field (#2060)** (605a9a1)
- **Fix dry run mode (#2027)** (ce1c0fc)
- **Fix the error message of maybe_enable_async_tp() (#2011)** (11d73a2)
- **[simplefsdp] fix region ac in zero2-style FSDP (#1970)** (02990b0)
- **Fix bugs in initial_load_in_hf when enable_weight_tying=true in Qwen3 (#1999)** (5ecc871)
- **Fix AC(compile(model)) by disabling Dynamo LRU cache (#1991)** (b6b2c2d)
- **fix set_determinism on single gpu (#1983)** (bb308da)
- **Fix link to FLUX model in README (#1978)** (2ea6197)
- **Fix how SimpleFSDP get the nD mesh (#1959)** (bc3021e)
- **fix setting ft state dicts when ft checkpointing is disabled (#1915)** (e150caa)
- **[FLOPs] Fix attention FLOPs estimate (#1923)** (5a090f2)
- **Fix import issue of gpt-oss (#1967)** (de4592d)
- **[Reland] Fix SimpleFSDP aot_eager_autobucket (#1965)** (3e084f4)
- **[compiler toolkit] Fix llama3 kwargs (#1954)** (f1b3c9f)
- **Fix the vlm test (#1949)** (91ee9ad)
- **[minor fix] job config (#1934)** (81a36c5)

### Backend
- **Remove psutil as part of requirements (#2181)** (81af883)
- **Bump torchtitan version to v0.2.1 (#2180)** (64b5e15)
- **Multiprocess simple RL loop (#2158)** (4b3d25a)
- **Bump tj-actions/changed-files from d6e91a2266cdb9d62096cebf1e8546899c6aa18f to e0021407031f5be11a464abee9a0776171c79891 in the github-actions group (#2167)** (1bd2548)
- **Replace `logger.warn()` to `logger.warning()` , allow `log_validation` to log `extra_metrics` and expose common wandb args (#2166)** (b786a3d)
- **Expose common dataloader args (#2097)** (658f94c)
- **Upgrade GitHub Actions for Node 24 compatibility (#2164)** (fd49b4b)
- **Integrate DeepEP to torchtitan (#2107)** (36a4b69)
- **[RELAND] Let CUDA and ROCm read different loss result (#2157)** (f64bbad)
- **Run vLLM inference using torchtitan model definition (single GPU) (#2119)** (e36d027)
- **Improve the loss_compare.sh logic (#2143)** (c283a84)
- **Upgrade GitHub Actions for Node 24 compatibility (#2151)** (f3748d8)
- **Upgrade GitHub Actions to latest versions (#2152)** (c1f4e94)
- **Implement ciflow/rocm on Torchtitan (#2114)** (9bc50ea)
- **Enable static type checking with Pyrefly (#2136)** (64dc922)
- **Enable PP and EP overlap for MoE (#1721)** (fcc5643)
- **Clarify contribution guidelines. (#2134)** (669845f)
- **Remove caching for attention masks (#2117)** (4389efd)
- **PyTorch depends on psutil (#2132)** (a632855)
- **Rename auto_parallel experiment to autoparallel (#2128)** (fbafd44)
- **[forge] allow torchforges to set checkpoint base folder (#2131)** (f3f2e8f)
- **[compiler toolkit] Disable CUDAGraph integration test (#2127)** (d192411)
- **Remove the hack for SAC + FlexAttention (#2118)** (575674a)
- **Enables parsing of --compile.components through CLI (#2115)** (e98ae99)
- **Update hf_assets_path for llama4 (#2110)** (1168f9e)
- **Mark input tokens to routed experts as dynamic to avoid a recompile (#2007)** (8d020cc)
- **Make sure log after distributed initialized. (#2102)** (571ce7c)
- **modify save list for varlen attn (#2082)** (53e949c)
- **Improve logging by formatting the dict as JSON. (#2094)** (befb7ae)
- **[Local Tensor] Replace dry_run.py with fake mode implementation (#2057)** (d0393b3)
- **skip varlen integration test on rocm (#2085)** (607c70d)
- **Autoparallel as an experiment in main (#2054)** (7e10d60)
- **Print out the version number (#2083)** (7e1edb6)
- **Update transformers backend name (#2075)** (ad9f188)
- **remove scatter_add in MoE implementation (#1974)** (e1f7f31)
- **adding variable length attention to llama3 8b   (#2000)** (f8fa21e)
- **3outeille/transformers backend (Dense model only) (#2048)** (58fa181)
- **Let loss_compare.py check the repo cleaness (#2062)** (8bf2265)
- **Re:Run Torchtitan ROCm workflow on cron schedule & push to Main branch only (#2018)** (4a5fa99)
- **[compiler toolkit] Port manual bucketing from SimpleFSDP experiment (#2056)** (bfdc974)
- **[compiler toolkit] Port joint_ac_pass from simplefsdp (#2051)** (3819737)
- **Print device and stride when print module (#2045)** (028a455)
- **[Flux] Update integration test badge in README.md (#2019)** (23c993c)
- **[Compiler Toolkit] Make compiler toolkit work with checkpoint (#2030)** (e7ee95a)
- **Revert PR-2016 & Redo "Run Torchtitan ROCm workflow on cron schedule & push to Main branch only" (#2017)** (55c63c1)
- **Run Torchtitan ROCm workflow on cron schedule & push to Main branch only (#2016)** (2f9b44d)
- **[easy] [compiler toolkit] Clean up unused function (#2014)** (edbf349)
- **set pg names (#1986)** (20fcfd7)
- **[compiler toolkit] specify passes through config (#2006)** (f4514ef)
- **[RFC] Seperate init_distributed_env from the Trainer.__init__ (#2003)** (61c25f8)
- **Deduplicate TorchTitan main function (#1995)** (4caa379)
- **[TorchComms + titan] Update ReadME for torchComms repo (#1992)** (176498c)
- **[compiler toolkit] Prepare deepseek to accept graph passes (#1982)** (8659543)
- **Reduce CI time (#1948)** (60f8e55)
- **[Flux] Enable unique random seed for multiple ranks (#1946)** (ff07852)
- **Improve SimpleFSDP typing and remove a finished TODO (#1960)** (a3e170c)
- **Second version of degub/deterministic configs. (#1807)** (e61f2cc)
- **[MoE][compile][full ac] weave torch.compile around the FSDP(GroupedExperts) graph break (#1895)** (2a7a148)
- **[compiler toolkit] Code deduplication between llama3 and deepseek_v3 (#1958)** (89c631c)
- **[Compiler Toolkit] Apply autobucketing_reordering_pass (#1951)** (f1a96b3)
- **[Compiler Toolkit] Avoid DTensorize BlockMask for FlexAttention (#1952)** (db47630)
- **Remove the unused compiled_autograd option (#1939)** (8228c08)
- **Graduate flux from experiment folder to core torchtitan (#1858)** (7f126cb)
- **[Compiler Toolkit] JointGraph-based Training Prototype for llama3 (#1794)** (06ec495)
- **[compiler toolkit] Enable kwargs input for deepseek_v3 (#1942)** (f51ab9f)
- **[Reland] Setup Compiler Toolkit  experiment folder for dsv3 (#1906) (#1938)** (d31d7e7)
- **[misc] pyproject.toml, unused var in train.py (#1933)** (10d694b)
- **[TorchComms] Enable CP in CI test (#1928)** (b01308a)
- **gpt-oss model enablement (#1754)** (de335b3)
- **Update PP to release memory earlier (#1922)** (e5ef99a)
- **[TorchComms] integration CI tests (#1927)** (b1644a4)
- **Allow to specify the file name to save config (#1921)** (7b96efc)
- **[TorchComm] Remove async TP and CP from known issue list (#1926)** (d6928f1)
- **Check torch.compile numerics in simpleFSDP tests (#1925)** (025c21b)
- **allow saving config to file (#1904)** (755ce8f)
- **[Refactor] Rename datasets to prepare for multimodal datasets (#1916)** (a8899e4)

_Recap by [Repo Wrapped](https://repowrapped.com/gh/pytorch/torchtitan?utm_source=github-action)._