microsoft/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

DeepSpeed trained the world's most powerful language models (MT-530B, BLOOM) Azure and DeepSpeed empower easy-to-use and high-performance model training. DeepSpeed is an important part of Microsoft’s new AI at Scale initiative to enable next-generation AI capabilities at scale. It is not tied to specific PyTorch or CUDA versions.