NVIDIA DGX Cloud Introduces Ready-To-Use Templates to Benchmark AI Platform Performance

In the rapidly evolving landscape of AI systems and workloads, achieving optimal model training performance extends far beyond chip speed. It requires a comprehensive evaluation of the entire stack, from compute to networking to model framework. 

Navigating the complexities of AI system performance can be difficult. There are many application changes that you can make, from precision to parallelism, but they currently require significant effort and specialized knowledge to implement effectively. 

NVIDIA DGX Cloud Benchmarking Recipes offer guidance for boosting training performance by sharing what good performance looks like per workload and best practices for how to get there. 

Fore more information, see NVIDIA DGX Cloud Benchmarking Recipes.

Evaluating an AI system holistically

DGX Cloud Benchmarking Recipes are an end-to-end benchmarking suite that can both measure performance in real-world environments and identify optimization opportunities in AI training workloads. These benchmarking recipes are crafted to evaluate performance for real-world AI applications, factoring in the complete AI stack.

Chip-centric metrics, such as peak floating-point operations per second (FLOPS) and bandwidth, can be inadequate in estimating end-to-end performance. Traditionally, peak FLOPS has often been used to compare platforms, but FLOPS is only one of the many components that affect the end-to-end application performance. 

In practice, the training time for an AI model is a function of many other components, such as network, software, firmware, and underlying infrastructure. 

For example, the high-bandwidth NVIDIA NVLink network fabric enables scaling parallelism strategies, such as tensor parallelism, beyond the traditional single-server, 8-GPU limit.

With NVIDIA Grace class systems, the NVLink networking layer enables you to achieve higher FLOPS in real-world applications, bridging the gap between theoretical and practical performance.

Evaluating AI platforms solely through FLOPS can result in inaccurate estimations of total training time and the associated costs, without accounting for the rest of the platform.

For modern AI workloads such as fine-tuning Llama 3.1 models, it’s more accurate to use benchmarks that measure end-to-end performance across the entire system, providing a holistic view of how a platform will perform in actual usage scenarios. 

Infrastructure factors affecting performance include the following:

• Server hardware designs
• Operating systems
• Virtualization layers
• Software stacks
• Network architectures
• Storage implementations

AI workload factors affecting performance include the following:

• Compute-to-communication ratio
• Model scaling factors
• Batch size
• Precision format
• Data loading strategies

Tuning workloads to optimal performance 

Beyond the job execution aspect of benchmarking, NVIDIA DGX Cloud Benchmarking Recipes are also playbooks for optimizing popular models and workloads. These recipes provide workload-specific strategies to maximize performance for popular models such as Llama 3.1, Grok, and Mixtral. 

Workload	Type	Description	Container Version	Dataset	Max Scale (#GPUs)	DTYPE
Nemotron4	Training	15B and 340B benchmarks	24.09	Synthetic	2048	FP8, BF16
Nemo Megatron	Training	175B benchmarks	24.05	Pile	2048	FP8, BF16
Llama 3.1	Training	8B, 70B, and 405B benchmarks	24.09	Pile	2304	FP8, BF16
PaXML	Training	5B and 175B benchmarks	24.03.04	Synthetic	2048	FP8, BF16
Maxtext	Training	Llama2 70B benchmarks	2024.12.09	Synthetic	2048	FP8, BF16
Grok1	Training	Grok1 314B benchmarks	24.09	Synthetic	2048	FP8, BF16
Llama 2	Fine Tuning	Hugging Face 70B benchmarks	24.02	HF Llama2	512	BF16
Mistral	Fine Tuning	Hugging Face 7B benchmarks	24.02	HF Mistral	256	BF16

Table 1. Available workloads in the DGX Cloud Benchmarking Recipes catalog

In Table 1, workloads include both training and fine tuning and support both FP8 and BF16 where possible.

Each training workload has a different fingerprint of how it exercises the platform. A basic question you might ask about a workload’s fingerprint is, “How much does compute time overlap with communication or networking time?” 

Some models may be more compute-bound and some are more communication bound, depending on the choice of parallelism and hyperparameters such as sequence length and batch size. Scaling behavior also varies between models as the number of GPUs are increased, as well as type of scaling, weak or strong.

For each workload and cluster scale, you must tune your model and system to achieve optimal performance. 

On the model side, this may involve adjusting the parallelism strategy, batch sizes, precision formats, and data loading strategies, among other configurations.

On the system side, make sure that the workload makes maximal use of NVLink high bandwidth (for example for tensor and context parallelism) and confirm that the scale-out fabric is not a blocker for the corresponding networking collectives (for example for pipeline or expert parallelism). 

The latter requires a fabric that provides low transport latency (RDMA) and effective congestion management and adaptive routing as found in the reference NVIDIA SpectrumX and InfiniBand networking architectures.

For efficient scaling of AI workloads, using these technologies is essential as they help mitigate the impact of jitter, ensuring consistent performance and reliability. 

Using FP8

DGX Cloud Benchmarking Recipes provide optimized configurations and tuning recommendations specifically for FP8 workloads, helping you achieve optimal performance with this precision format. For example, the recipe for Llama 3.1 70B training includes FP8 settings that have been carefully tested and optimized for DGX Cloud platforms.

Understanding what constitutes good performance for a given AI workload can be complex. DGX Cloud Benchmarking Recipes provide a range of baseline performance results for various popular models, enabling you to set realistic expectations and goals for your own implementations. 

These baselines include metrics such as model FLOPS utilization (MFU), which measures how efficiently a model uses the available compute resources. You can see how MFU and throughput compare for popular models. By comparing your results to these benchmarks, you can gauge the effectiveness of optimizations and identify areas for improvement.

DeepSeek-R1 is a 671B-parameter model that runs on one NVIDIA H200 GPU node. Its high compute utilization shows how holistic optimization of compute, networking, and parallelism strategies can push delivered performance closer to theoretical limits.

Systematic benchmarking enables direct comparisons, helping teams collaboratively optimize models and platforms to maximize the value of GPU systems.

Finally, these per-workload performance optimizations also show the need for further research and discussion around application tuning. For example, the recommended usage of the parallelization strategies varies across combinations of workloads and platforms.