PyTorch offers insights into deep learning, neural network modeling, and machine learning research, providing documentation, tutorials, and best practices for building and training models with PyTorch framework. By exploring PyTorch's curated content, developers can learn about tensor computations, autograd mechanisms, and model deployment strategies for solving complex problems in computer vision, natural language processing, and reinforcement learning. Whether you're a researcher, practitioner, or enthusiast, PyTorch offers resources to advance your understanding of deep learning and push the boundaries of AI innovation.

PyTorch

A joint PyTorch and Nebius experiment demonstrates up to 41% faster pre-training of DeepSeek-V3 MoE models on a 256-GPU NVIDIA B200 cluster using TorchTitan. Two orthogonal optimizations were evaluated: MXFP8 mixed-precision training (via TorchAO) and DeepEP communication acceleration. For the 671B model, DeepEP alone yielded +32% throughput over BF16 baseline by replacing standard all-to-all with GPU-initiated RDMA+NVLink kernels. Combining MXFP8 on grouped GEMMs with DeepEP pushed total throughput to 918 tokens/sec, a +41% gain. Convergence experiments on the 16B model confirmed MXFP8 training matches BF16 quality with no degradation over 1,500 steps. Future work includes enabling MXFP8 on linear layers once torch.compile supports MXFP8 with tensor parallelism.

Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan – PyTorch

Experiment 2: DeepSeek-V3 16B MoE Loss Convergence Validation