NVIDIA Blackwell Shatters Performance Records in MLPerf Training v5.0 Benchmarks

The race to build state-of-the-art large language models (LLMs) demands unprecedented computational power. As models scale to hundreds of billions of parameters trained on trillions of tokens, high-performance systems become critical to achieving breakthroughs while managing training timelines. NVIDIA’s new Blackwell architecture demonstrates remarkable gains in the latest MLPerf Training v5.0 benchmarks, setting new records across seven key AI workloads.

Benchmark Dominance

NVIDIA’s Blackwell-based systems swept all MLPerf Training v5.0 categories:

Workload	Model	Training Time (min)
LLM Pretraining	Llama 3.1 405B	20.8
LLM Fine-Tuning	Llama 2 70B-LoRA	0.56
Text-to-Image	Stable Diffusion v2	1.04
Graph Neural Network	R-GAT	0.84
Recommender System	DLRM-DCNv2	0.7
Natural Language Processing	BERT	0.3
Object Detection	RetinaNet	1.4

Results from MLCommons.org (June 4, 2025)

Architectural Leap: Blackwell vs. Hopper

The GB200 NVL72 rack-scale system—featuring 72 Blackwell GPUs interconnected via fifth-generation NVLink—delivered unprecedented performance improvements over the previous Hopper architecture:

• 2.2x faster Llama 3.1 405B pretraining at 512-GPU scale
• 2.5x faster Llama 2 70B fine-tuning at 8-GPU scale
• 2.6x faster Stable Diffusion v2 training
• 2.25x faster graph neural network training

These gains stem from Blackwell’s architectural innovations:

• Second-generation Transformer Engine
• 9x larger NVLink domains
• HBM3e memory with higher bandwidth/capacity
• Enhanced FP8 compute capabilities

Key Technical Optimizations

1. Memory Efficiency Innovations
CUDA Graphs were reengineered to process full forward-backward passes per GPU, reducing memory footprint by enabling cross-layer memory reuse. This also minimized CPU overhead—critical when scaling to thousands of GPUs.

2. Communication-Compute Overlap
Optimized GEMM and communication concurrency was achieved through:

• CUDA stream prioritization for communication kernels
• Copy-engine implementations for tensor parallelism collectives
• Pipelined reduce-scatter and AllReduce operations

3. Model Parallelism Breakthroughs
Megatron-Core introduced “DP-Last” (TP-CP-PP-DP) mapping specifically optimized for GB200 NVL72 topology, demonstrating near-linear scaling efficiency for the 405B-parameter Llama model across 2,496 GPUs.

4. Kernel-Level Enhancements

• Flash Attention kernels reduced register spills in backward passes
• Apex GroupNorm kernels minimized memory footprint
• RMSNorm kernels accelerated layer normalization
• FP8 storage for SwiGLU inputs conserved memory

Real-World Impact

These advancements translate to tangible benefits:

• Fine-tuning acceleration enables organizations to deploy custom models faster
• Memory optimizations allow fitting Llama 2 70B entirely on a single GPU
• Scaling efficiency reduces time-to-solution for massive models

Looking Ahead

NVIDIA’s benchmark submissions demonstrate Blackwell’s architectural advantages for next-generation AI workloads. As MLPerf evolves, these innovations provide the foundation for training increasingly sophisticated models that push the boundaries of AI capabilities.