AI and scientific computing applications are excellent examples of distributed computing problems. These problems are too large and compute-intensive to run on a single machine. They are broken down into parallel tasks, which are then distributed across thousands of computing engines, such as CPUs and GPUs.

To achieve scalable performance, the system relies on distributing the workload across multiple nodes, like training data, model parameters, or both. Then, these nodes must frequently exchange information, such as gradients computed during backpropagation in model training, necessitating efficient collective communications like all-reduce, broadcast, as well as gather and scatter operations.

These collective communication patterns ensure the synchronization and convergence of model parameters across the entire distributed system. The efficiency of these operations is crucial for minimizing communication overhead and maximizing parallel computation, as poorly optimized collective communications can lead to bottlenecks, limiting scalability.

Bottlenecks arise from several factors：

Latency and Bandwidth Limitations: Collective operations rely on high-speed data transfer between nodes, which is constrained by the physical network latency and bandwidth. As the system scales up, the amount of data to be exchanged increases, making the time spent on communication more critical than computation.

Synchronization Overhead: Many collective operations require synchronization points where all participating nodes must reach the same state before proceeding. If some nodes are slower, the entire system experiences delays, leading to inefficiency, known as “stragglers.”

Network Contention: As more nodes attempt to communicate simultaneously, the network becomes more congested, and the competition for bandwidth and network resources increases, further slowing down collective operations.

Suboptimal Communication Patterns: Some collective communication algorithms (like tree-based reduction or ring-based all-reduce) are not always well-optimized for large-scale systems, resulting in inefficient use of available resources and increased latency.

Overcoming these bottlenecks requires improvements in network technology (e.g., InfiniBand or RDMA) and algorithm optimization (e.g., hierarchical all-reduce or pipelining techniques) to minimize synchronization delays, reduce contention, and optimize data flow between distributed systems.

Create NVIDIA SHARP

Key collective communications enable all computing engines to exchange data with each other. Managing such communication on NICs or servers requires exchanging large amounts of data and can be affected by latency or variations in collective performance, also known as server jitter.

By shifting the responsibility of managing and executing these collective communications to the switch fabric, the amount of data transferred can be halved, and jitter can be minimized. NVIDIA’s Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) is the technology that realizes this concept and introduces the concept of network computing. It is integrated into the switch ASIC and is designed to accelerate collective communications within distributed computing systems.

SHARP has been introduced with NVIDIA InfiniBand networks, allowing the offloading of collective communication operations (like all-reduce, reduce, and broadcast) from the servers’ computing engines to the network switches. By performing reduction operations (summation, averaging, etc.) directly within the network fabric, SHARP improves the performance of these operations and overall application performance.

Leveraging NVIDIA SHARP to drive technological progress

The first generation of SHARP was designed specifically for scientific computing applications, focusing on small message reduction operations. It was introduced with the generation of NVIDIA EDR 100Gb/s switches and quickly received support from leading Message Passing Interface (MPI) libraries. SHARPv1’s small message reduction supports multiple scientific computing applications in parallel.

MVAPICH2 is an open-source implementation of the MPI standard, designed for high-performance computing (HPC) environments. The team at The Ohio State University responsible for the MVAPICH MPI library has demonstrated the performance achievements of SHARP on the Frontera supercomputer at the Texas Advanced Computing Center. From a 5x performance improvement in MPI AllReduce to a 9x performance boost in MPI Barrier collective communications. For more information, see Scalable MPI Collectives with SHARP: Large-Scale Performance Evaluation on the TACC Frontera System.

The second generation of SHARP was introduced with NVIDIA HDR 200Gb/s Quantum InfiniBand switches and added support for AI workloads. SHARPv2 supports large message reduction operations, supporting a single workload at a time. This version further enhances the scalability and flexibility of the technology, supporting more complex data types and aggregation operations.

The June 2021 NVIDIA MLPerf submission and results demonstrated the performance advantages of SHARPv2, showing a 17% improvement in BERT training performance. For more information, see MLPerf v1.0 Training Benchmark: Insights into Record-Breaking NVIDIA Performance.

NVIDIA Vice President and Chief Architect of AI Systems, Michael Houston, introduced the AllReduce performance advantages of SHARPv2 in a Machine Learning Systems course at UC Berkeley.

The 2x performance advantage in AllReduce bandwidth can improve BERT training performance by 17%.

he University of California, Berkeley Machine Learning Systems course example.

word count: 10, token count: 18, tokens used: 4133, model: grok-beta(grok-beta), time: 18:01

Recently, the third generation of SHARP was introduced with the NVIDIA Quantum-2 NDR 400G InfiniBand platform. SHARPv3 supports multi-tenant network computing for AI workloads, meaning that multiple AI workloads can be supported in parallel compared to a single workload with SHARPv2.

Jithin Jose, Chief Software Engineer at Microsoft Azure, showcased the performance of SHARPv3 at the “Transforming the Cloud into Cloud-Native Supercomputing: Best Practices with Microsoft Azure” session. Jithin presented InfiniBand’s network computing technology on Azure and demonstrated a performance advantage of nearly an order of magnitude in AllReduce latency.

The AllReduce latency performance advantage of SHARPv3.

End-to-end AI system optimization

A powerful example of SHARP can be seen through the all-reduce operation. During model training, gradients are summed across multiple GPUs or nodes, and SHARP aggregates these gradients within the network, eliminating the need to send the entire dataset between GPUs or nodes. This reduces communication time, thereby speeding up AI workload iterations and increasing throughput.

Before the advent of network computing and SHARP, the NVIDIA Collective Communication Library (NCCL) communication software would copy all model weights from the graph, perform the all-reduce operation to compute the sum of weights, and then write the updated weights back to the graph, resulting in multiple data copies.

In 2021, the NCCL team began integrating SHARP, introducing user buffer registration. This allowed NCCL collectives to directly use pointers, eliminating the need for data copying back and forth during the process, thus improving efficiency.

Today, SHARP is tightly integrated with NCCL, which is widely used in distributed AI training frameworks. NCCL is optimized to offload critical collective communication operations to the network, fully leveraging SHARP to significantly enhance the scalability and performance of distributed deep learning workloads.

SHARP technology helps improve the performance of distributed computing applications. It is being used by high-performance computing (HPC) supercomputing centers for their scientific computing workloads and by AI supercomputers for AI applications. SHARP is the “secret sauce” for gaining a competitive advantage. A large service provider used SHARP to improve the performance of its internal AI workloads by 10% to 20%.

SHARPv4

SHARPv4 introduces new algorithms that support more types of collective communications, which are now utilized in leading artificial intelligence training applications. SHARPv4 will be released alongside the NVIDIA Quantum-X800 XDR InfiniBand switch platform, taking network computing capabilities to an even higher level.