## NVIDIA/Megatron-LM — v1.0…v2.0

_442+ commits._

### Features
- **Use new api to get loss scale when finetuning.** (a13cbe1)
- **add multi-tensor-apply to clip grad** (9a01031)
- **working on the new optimizer** (c6a58e4)
- **Add residual connection in fp32 machinery** (8bed1d6)
- **Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)** (62632d3)
- **Add comment describing _PIPELINE_GLOBAL_RANKS** (5131590)
- **Add implementation for pipelined zeroshot GPT-2 evaluation** (0c15163)
- **Add pipelining to GLUE and RACE tasks** (caa9dca)
- **Support for ramping up the batch size** (de0b70a)
- **Add constant num micro-batches calculator** (feecd5d)
- **Add micro-batch size calculator** (6ea2392)
- **New example scripts showing how to use MP, and some notes to main README** (a5a4192)

### Fixes
- **Merge branch 'TOC-fix' into 'main'** (be473a5)
- **TOC fix** (3173378)
- **found a bug in l2 norm calculation** (6191ff5)
- **Quick fix for pipeline tasks to get learning rate correct** (6e83649)
- **Fix TensorBoard writes** (b81cad6)
- **Fix loss addition in TensorBoard** (5a304ed)
- **Fix some bugs, add exit-duration capability** (a31833c)
- **Fix text generation without recompute** (1d4e876)
- **found a bug in consumed tokens initialization** (e2a4d42)
- **Merge branch 'fix/help-title-dist' of https://github.com/lazykyama/Megatron-LM into community-fixes** (356f877)
- **Merge branch 'patch-1' of https://github.com/sublee/Megatron-LM into community-fixes** (24ccd8a)
- **Merge branch 'patch-2' of https://github.com/nakosung/Megatron-LM into community-fixes** (cec3dbe)
- **Merge branch 'patch-1' of https://github.com/nakosung/Megatron-LM into community-fixes** (8d0da20)
- **Merge branch 'patch-1' of https://github.com/dweekly/Megatron-LM into community-fixes** (5e93404)
- **Merge branch 'patch-1' of https://github.com/akhileshgotmare/Megatron-LM into community-fixes** (642c644)

### Backend
- **Merge branch 'readme_update_jan_2021' into 'main'** (4635bdf)
- **Readme update + change gpt2 to gpt** (152aab3)
- **Merge branch 'validation_loss_reporting' into 'main'** (86eb5bd)
- **fixed validation loss reporting in tensorboard** (5b74f76)
- **Merge branch 'default_args' into 'main'** (f5eac3d)
- **Change some arguments to default to on.** (9acc895)
- **Merge branch 'optimize_refactor' into 'main'** (9a29754)
- **Merge branch 'optimize_refactor_backward_compat' into 'optimize_refactor'** (42c6d8e)
- **fixed assert issue in clip grads for none grads** (82945c1)
- **added comment about norm power** (571f10a)
- **addressed Jareds and Deepaks comments** (db88a27)
- **addressed jareds comments** (7417078)
- **Merge branch 'task_head_fix' into 'main'** (b69bc7e)
- **Merge branch 'optimize_refactor_finetune_fix' into 'optimize_refactor'** (512337f)
- **Only create task heads on last pipeline stage.** (f772fbc)
- **Merge branch 'optimize_refactor_reload_model_params' into 'optimize_refactor'** (7381754)
- **Merge branch 'send_and_recv' into 'main'** (6fa3684)
- **some small fixes** (9eedf89)
- **added backward compatibility** (203235f)
- **further refactor, tested, and changed master to main** (983cc31)
- **further refactor, tested, and changed master to main** (22fa9ba)
- **added reload model params for finetuning** (160ba68)
- **Use batched send and recv instead of torch.distributed.ring_exchange()** (d899988)
- **addressed deepaks comments** (43529f7)
- **moved compile helper to initialize** (242770d)
- **added clip grads** (a495871)
- **done with refactoring and checks** (b77d906)
- **moved grad scaling to multi-tensor-apply** (345f5d0)
- **further refactor, matching old results** (e68ea21)
- **slight refactor of clip grads** (b0a3fdf)
- **moved module to model and removed fp16** (b84d7a9)
- **load and save state dicts added** (97ba5c0)
- **further refactoring** (0888a3e)
- **grads is removed from mpu** (dfd8ed4)
- **clip grad fixed and moved to optimizer** (4687967)
- **moved entire optimizer build and tested** (28062e1)
- **megatron optimizer tested, before working on clip grad** (fb218c9)
- **fp32 is also working** (2eaa3cc)
- **Merge branch 'validation_loss_after_checkpoint' into 'main'** (2348c99)
- **Checkpoint should be saved only after evaluation pass is run to make sure validation losses are identical after loading checkpoint** (13bde16)
- **Merge branch 'fp32_residual_conn' into 'main'** (c84f075)
- **Address Deepak's comments** (83671bb)
- **Merge branch 'pipeline_parallel_main' into 'main'** (9b174da)
- **Added a comment to justify 80 percent** (25c07e1)
- **Move args writer to the beginning of training** (6e9d5cb)
- **Cleaned up load/save checkpoint printing** (8a6e56b)
- **Some bugfixes** (29a6954)
- **Last epoch should not be globally shuffled** (3918111)
- **Address Jared's comments** (56243e1)
- **Nicer error messages for deprecated arguments** (2623551)
- **Change lr-warmup-percent to lr-warmup-fraction** (9321d5c)
- **Work batch-size name changes into task code** (3afcba6)
- **Initial implementation of pipelined text generation** (5c45db4)
- **Better memory tracking across pipeline-parallel ranks** (3574b8e)
- **Address Jared's comments** (00ac56a)
- **Sample based learning rate computation** (22ab91b)
- **Minor fixes for batch size rampup** (6a68502)
- **Minor refactoring** (c30ba0f)
- **Rename --batch-size to --micro-batch-size and drop in-minibatch from --num-micro-batches-in-minibatch** (9019bbf)
- **Make an eval iteration the same number of samples as a training iteration** (a84a5fa)
- **Merge branch 'main' into pipeline_parallel_merge** (2cf1d6d)
- **Merge branch 'consumed_tokens_restart_fix' into 'main'** (3aacd95)
- **Merge branch 'main' into pipeline_parallel_main** (91d4a60)
- **Merge branch 'megatron_sampler' into 'main'** (75bd9b5)
- **Merge branch 'blendable_dataset' into 'megatron_sampler'** (fac6718)
- **Merge branch 'refactor_learning_rate' into 'blendable_dataset'** (1eda0a1)
- **addressed Jareds comments** (fa80af2)
- **Merge branch 'blendable_dataset' into refactor_learning_rate** (4550454)
- **addressed Jareds comments** (9898969)
- **Merge branch 'megatron_sampler' into blendable_dataset** (bc56e4a)
- **addrressed jareds comments** (cebd3b8)
- **refactored learning rate scheduler so addition of variable batch size is easier** (ff12df6)
- **added refactored learning rate** (1619361)
- **implemented blending datasets** (6529003)
- **Merge branch 'megatron_sampler' into blendable_dataset** (9a0808c)
- **added blendable dataset** (d3bb1a0)
- **added consumed tokens to checkpoints and some refactoring** (f0a445f)
- **simplified sampler** (4311b69)
- **Merge branch 'main' into pipeline_parallel_main** (63c340e)
- **Merge branch 'update-norm' into 'main'** (ea81d62)
- **Merge branch 'community-fixes' into 'main'** (ac837a4)
- **Replace deprecated torch.norm with torch.linalg.norm.** (17035d6)
- **Merge branch 'main' into pipeline_parallel_main** (6fae152)
- **Merge branch 'finetune_fix' into 'main'** (e7c7a78)
- **Update code used for finetuning to latest API.** (b219ff0)
- **Make sure dataloader state is the same after checkpoint is loaded** (cd4822f)
- **Move division of loss tensor by number of microbatches to training.py** (c671de3)
- **Small bugfix in bert_model.py: make sure word_embeddings is initialized before instantiating lm_head** (69a546b)
- **Refactor code according to Jared's comments: move pipelining and non-pipelining training loops into separate methods** (1979c24)
- **Allocate tensor in `communicate()` method directly on GPU (instead of allocating on CPU and then moving to GPU)** (9ff6f47)
- **Remove timing labels that don't make sense** (fdcaeba)
- **Better communication timing** (7d367b1)
- **Divide gradient by number of microbatches in minibatch** (3d7194c)
- **Better 'forward' and 'backward' timing in megatron/training.py** (a6756bf)
- **Move training schedule to 1F1B for memory efficiency** (3e6898e)
- **Only transpose hidden_states when necessary** (6abf39b)
- **Refactor word_embeddings_weight() logic into separate method, and other Mohammad comments** (57c3b36)
- **Log times for various sub-operations in forward and backward pass in main training loop** (eed0062)
- **Throw exception if ring_exchange is not available when pipeline_model_parallel_size > 1** (2d8de29)
- **Bugfix in megatron/training.py: correct global_batch_size computation** (7ce373f)
- **Improve names of identifiers used for timing in main training loop** (9d4c735)
- **Clarifications in comments and minor refactoring to make main training loop more readable** (8fb2bc8)
- **Remove unused parameter sharing logic** (1271fd7)
- **Bugfix in main training loop: Update master_grads only after grads are correctly accumulated** (9b55856)
- **Simplify logic in megatron/fp16/fp16.py** (767e6e9)
- **Small notes in comments in response to Jared's comments** (aa9cae2)
- **Address Jared's comments in README and loss_scaler.py** (dd07940)
- **Improve time logging when num_microbatches_in_minibatch > 1** (6374022)
- **Back compatibility of checkpoints: use `model_parallel_size` when checking for equality of args** (d5b526d)
- **Refactor communication code in main training loop to helper method** (318d68c)
- **Back compatibility of checkpoints: don't rename model_parallel_rng_tracker** (e805f0b)
- **Removal of unneeded changes so that diff is smaller** (275d4e6)
- **Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP** (52a5f2f)
- **Pipeline parallelism implementation with periodic full-pipeline syncs** (7abd3e9)
- **fp32 working** (28cd66e)
- **Merge branch 'fix_logging' into 'main'** (b4b0d73)
- **fixed loss average when all but one value is skipped** (664cd28)
- **Change distributed argument group title to appropriate one.** (c147a3e)
- **Merge branch 'main_evaluate_wiki' into 'main'** (79888e1)
- **fixed wiki evaluation issue** (ef2adb5)
- **fixed wiki evaluation issue** (38c45de)
- **Merge branch 'main_beta' into 'main'** (64cf3d9)
- **added adam betas and eps as arguments** (c55e154)

_Recap by [Repo Wrapped](https://repowrapped.com/gh/NVIDIA/Megatron-LM?utm_source=github-action)._