Megatron-LM

https://github.com/NVIDIA/Megatron-LM This is probably a good baseline to start thinking about LLM training at scale.

The known SOTA for GPU flops utilisation for training on that many GPUs is somewhere between 50-60%, e.g. https://github.com/NVIDIA/Megatron-LM . If they really managed to get 95%, it's a huge advance in the state of the art. I guess by training time they meant with respect to downtime of the hardware, not utilisation of the hardware potential.

https://github.com/NVIDIA/Megatron-LM > Megatron is a large, powerful transformer [...]

The pipedream-2bw paper[1] and the zero-offload paper[2] both show that 1-step delayed asynchronous gradient update doesn't affect the convergence (and perplexity) while improve the training efficiency (by fully utilize the bubbles in pipeline parallelism) at a large margin. However, both the Megatron-LM[3] and the DeepSpeed[4] don't use pipedream-2bw scheduling. Could anyone share me some insights or ideas about why such an efficient scheduling scheme doesn't get popular in the LLM pretraining community? Does it suffer convergence/accuracy issue in practice? Or are there any other concerns that blocking it become the default / most popular pipeline parallelism scheduling? [1]: https://arxiv.org/abs/2006.09503 [2]: https://arxiv.org/abs/2101.06840 [3]: https://github.com/nvidia/Megatron-LM [4]: https://github.com/microsoft/DeepSpeed/issue

GPU cluster scaling has come a long way. Just check out the scaling plot here: https://github.com/NVIDIA/Megatron-LM

I'm very bullish on the entire sector. One incumbent vs startup story to watch in the AI accelerator space is NVidia vs Lightmatter, If they can realize the cost savings of photonic computing it looks like a 5-7x improvement. NVidia's Megatron trillion parameter language model requires astounding compute capabilities: 3000+ A100 GPUs, And while I don't see GPU dominance retreating through 2024 at least, as we get into universal translation and global parallel corpora by the end of the decade, the limits become apparent. And it probably won't be talent, design or money that becomes the bottleneck. But the relative difficulty of working with photonic crystals compared to the low hanging fruit of silicon that has provided such a bounteous harvest for the last 70 years. https://github.com/NVIDIA/Megatron-LM

Hi, author here! Some details on the model: * Trained 17GB of code from the top 10,000 most popular Debian packages. The source files were deduplicated using a process similar to the OpenWebText preprocessing (basically a locality-sensitive hash to detect near-duplicates). * I used the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) code for training. Training took about 1 month on 4x RTX8000 GPUs. * You can download the trained model here: https://moyix.net/~moyix/csrc_final.zip and the dataset/BPE vocab here: https://moyix.net/~moyix/csrc_dataset_large.json.gz https://moyix.net/~moyix/csrc_vocab_large.zip Happy to answer any questions!