Frontier model pre-training has converged on mixture of experts (MoE), which is fundamentally changing what limits large-scale AI training. As compute per token falls, communication increasingly determines how efficiently models scale across thousands of GPUs. NVIDIA GB300 NVL72 set a world record for pre-training DeepSeek-V3 671B at 1,648 TFLOPs per GPU, showing how advances across the entire AI platform—from silicon to networking to software—continue to push training performance forward. Every gain in pre-training efficiency means researchers can train larger models, run more experiments, and reach frontier capability faster on the same NVIDIA infrastructure.  

The rapid industry shift toward these MoE architectures is driven by their massive computational efficiency. Unlike dense models, where every token activates every parameter and compute per token rises with the total parameter count, MoE models activate a subset of parameters for each token. For example, DeepSeek-V3 holds 671B parameters but activates only ~37B parameters per token, reaching frontier scale at the per-token cost of a far smaller model.

The tradeoff is communication. Those experts live on other GPUs, so every MoE layer must dispatch each token to its experts and gather the results through an all-to-all communication pattern in both the forward and backward passes. This collective sits in the critical path, making throughput as dependent on communication as on compute. Because it occurs at every layer in every training step, small delays compound until the all-to-all can no longer be hidden behind compute, and adding GPUs no longer increases throughput.

MoE training step showing forward pass, backward pass, gradient sync, and optimizer step. Within each layer, attention connects to experts through tensor-parallel all-reduce and all-to-all communication. Per-layer communication is on the critical path; gradient sync occurs once per step and can overlap with compute.
Figure 1. Anatomy of an MoE training step, showing where communication happens and how it affects performance

This is why pre-training needs a tightly coupled scale-up domain, where every GPU can communicate with every other GPU over a non-blocking fabric that provides high, uniform bandwidth, low latency, and full bisection bandwidth as the domain grows. Training a model of this size requires more GPUs than a single domain can hold, and many domains must be linked together. This scale-out traffic is lighter and less frequent, but it must still complete within the compute window and remain predictable, so no single slow link governs the step. The challenge is two-tiered, and success is measured in delivered FLOPs, not peak FLOPs.

NVIDIA GB300 NVL72: Built for tightly coupled AI pre-training

A two-tiered communication challenge calls for a system designed around it, not just a faster chip. Compute, scale-up interconnect, scale-out networking, infrastructure processing, and software each carry part of the load. A shortfall in any one caps the whole.  GB300 NVL72 addresses these challenges together, a rack-scale system built through extreme co-design, where silicon, interconnect, networking, and software are engineered as one platform rather than assembled from parts. 

At its core is NVIDIA NVLink, the scale-up fabric that makes 72 NVIDIA Blackwell Ultra GPUs work as one. Fifth-generation NVLink gives each GPU 1.8 TB/s of bandwidth and 130 TB/s of non-blocking, all-to-all bandwidth across the rack, so every GPU reaches every other in a single hop. 

The width is only half of it; the path is the rest. NVLink is memory-semantic: a GPU reads and writes a peer’s HBM directly, as native load and store operations, over a lossless, flow-controlled fabric. The transfer is a hardware memory operation rather than a software send, so nothing in the data path adds latency, and reductions can run inside the switch as the data flows through. 

This is what the per-layer traffic demands: the tensor-parallel all-reduce and the MoE all-to-all stay inside the rack, at full bandwidth and low latency. Beyond the rack, the platform scales out over NVIDIA ConnectX-8 SuperNICs at 800 Gbps per GPU, with NVIDIA Quantum-X800 InfiniBand or NVIDIA Spectrum-X Ethernet, to keep the gradient traffic hidden behind the compute. 

Comparison of GPU-to-GPU memory access paths. NVLink scale-up connects a GPU to peer HBM through an NVLink Switch in two low-latency hops. Traditional scale-up routes through PCIe, NICs, and a packet-fabric switch across six stages, adding higher latency.
Figure 2. NVLink fabric compared to a traditional fabric

Co-design extends beyond the training fabric into infrastructure services and software. In production AI factories, NVIDIA BlueField data processing units (DPUs) provide an isolated infrastructure-processing domain for virtual networking, storage access, security, telemetry, and lifecycle management, reducing host CPU overhead for large-scale training jobs.

Training software covers both sides of the ecosystem. NVIDIA Megatron Core is built and tuned for these GPUs. NVIDIA has also actively contributed to open source frameworks to ensure TorchTitan and JAX run at full speed on GB300 NVL72 systems.

Exceptional pre-training performance with Megatron Core  

On the DeepSeek-V3 671B model, using 256 GPUs, Megatron Core reaches 1,648 TFLOPs per GPU on GB300 NVL72, compared to 606 TFLOPs per GPU on earlier GB200 NVL72 results, ~3x higher delivered throughput per GPU in a single generation. 

Bar chart comparing delivered training performance in TFLOPs per GPU on DeepSeek-V3 671B at 256 GPUs. GB200 NVL72 reaches 606 TFLOPs/GPU and GB300 NVL72 achieves 1,648 TFLOPS/GPU 3x higher performance for the same GPU count.
Figure 3. 3x higher delivered performance GB200 NVL72 (early software version) vs GB300 NVL72 (latest software version)

NVIDIA continues to optimize its software for better performance from the overall platform. On a pre-training workload the size of DeepSeek-v3 671B, those gains compound. On the same GB300 NVL72 rack-scale system, performance increased 1.5x in six months, driven by software improvements. This shows that raw performance and training throughput continue to improve well after the silicon ships. 

Bar chart showing delivered training performance in TFLOPs per GPU on DeepSeek-V3 671B at 256 GPUs, measured on unchanged GB300 NVL72 hardware. The performance rose from 1,088 6 months ago to 1,648 today, about 1.5 times higher, driven entirely by software optimization.
Figure 4. 1.5x higher performance in six months from software alone on GB300 NVL72

Leading pre-training frameworks accelerated by NVIDIA  

NVIDIA engineers contribute directly to the open frameworks the AI community relies on, adding optimizations that make them faster on NVIDIA GPUs. Developed collaboratively with the PyTorch and JAX communities, these contributions land continuously and improve performance over time.

TorchTitan is PyTorch’s native training stack, and NVIDIA contributions continuously improve its performance on GB300 NVL72. On DeepSeek-V3 671B, these optimizations compound to deliver approximately 6x higher delivered performance on the same infrastructure.

Bar chart of TorchTitan delivered performance evolution in TFLOPs per GPU on DeepSeek-V3 671B on GB300 NVL72. Across successive optimization stages, performance rises by about 6x overall, driven by NVIDIA software contributions to TorchTitan framework.
Figure 5. GB300 NVL72 performance evolution with TorchTitan optimizations 

JAX follows the same arc. Over six months, NVIDIA JAX optimizations have lifted performance improvements by nearly 10x on DeepSeek-V3 671B at 256 GPU scale—all from software optimizations. The latest software version reaches an exceptional performance throughput of 1,025 TFLOPS/GPU, and software optimizations continue to evolve from there.  

Bar chart of JAX delivered performance evolution  in TFLOPs per GPU on DeepSeek-V3 671B at 256 GPUs on GB300 NVL72, from January 2026 to today. Performance rises by nearly 10x over six months, driven by NVIDIA software optimization. 
Figure 6. GB300 NVL72 performance evolution with JAX optimizations

Scaling DeepSeek-V3 671B pre-training from 256 to 1,024 GPUs, Megatron Core holds 98.5% of its per-GPU performance while TorchTitan and JAX each hold 97%, so nearly all the added infrastructure turns into added system-level tokens per second throughput. That efficiency is the scale-out fabric doing its job efficiently as racks are scaled out with 800 Gb/s per-GPU networking; gradient traffic stays hidden behind compute, ensuring that adding more GPUs strictly increases total system throughput instead of bogging the network down with communication overhead. 

Chart of Megatron Core, TorchTitan, and JAX scaling efficiency for pre-training DeepSeek-V3 671B on GB300 NVL72. All three start at 100% at 256 GPUs and stay high as the count grows to 512 and 1,024, where Megatron Core holds 98.5% and TorchTitan and JAX each hold about 97%.
Figure 7. Scaling DeepSeek-V3 671B pre-training on GB300 NVL72. Megatron Core, TorchTitan, and JAX all hold per-GPU throughput, while scaling from 256 to 1,024 GPUs

Setting a world record

Pre-training DeepSeek-v3 671B with 256 GPUs achieves a world record 1,648 TFLOPs per GPU with GB300 NVL72, enabling the same training job to reach the same performance with a fraction of the hardware compared to the previous generation. 

The open frameworks tell the same story. Performance continues to improve on the same platform as the software evolves. These results are not the ceiling—they come from a platform whose hardware, interconnect, and software are designed together and continuously optimized. The record-setting performance of today is only the foundation for even higher performance tomorrow. 

Get started training frontier models on NVIDIA AI infrastructure

Acknowledgements

Setting a world record on the NVIDIA GB300 NVL72 platform by pre-training the DeepSeek-V3 model reflects the work of many exceptional engineers across the company. We would like to acknowledge the contributions of the following individuals (sorted by last name):

Aidyn Aitzhan, Michael Andersch, Jan Bernloehr, Santosh Bhavani, Ben Cashman, Carlo del Mundo, Ashraf Eassa, Fabio Paes Leme Ferriani, Matt Frank, Abhinav Goel, Vivek Goel, Elfie Guo, Eric Harper, Munira Hussain, Tomasz Jakubek, Masaki Kozuki, George Kurian, Himangshu Lahkar, Guihong Li, Kibibi Moseley, Nitin Nitin, Devin O’Kelly, Christian M. Sarofeen, Priya Sethuraman, Tejash Shah, Franciszek Szarwacki, John Tran, Qiyu Wan, and Cliff Woolley.