Unlock GPU accelerated RDMA with NVIDIA DOCA GPUNetIO

NVIDIA DOCA GPUNetIO is a library in the NVIDIA DOCA SDK specifically designed for real-time inline GPU packet processing. It combines technologies such as GPUDirect RDMA and GPUDirect Asynchronous to create GPU centric applications, where the CUDA core can directly communicate with network interface cards (NICs) for sending and receiving data packets, bypassing the CPU and excluding it from the critical path.

The core principles and applications of DOCA GPUNetIO have been discussed in previous articles such as “Inline GPU Packet Processing with NVIDIA DOCA GPUNetIO” and “Realizing the Power of Real Time Network Processing with NVIDIA DOCA GPUNetIO”, as well as in the DOCA GPUNetIO programming guide.

Previously, DOCA GPUNetIO, along with DOCA Ethernet and DOCA Flow, was limited to handling packet transmission on the Ethernet transport layer. With the release of DOCA 2.7, there is now a set of extended APIs that enable DOCA GPUNetIO to directly support RDMA communication from GPU CUDA cores using RoCE or InfiniBand transport layers.

The acronym RDMA describes a protocol that allows remote direct memory access from one computer’s memory to another computer’s memory, without involving any operating system in either computer. Examples of operations include RDMA writing and RDMA reading. It cannot be confused with GPUDirect RDMA, which is unrelated to the RDMA protocol. GPUDirect RDMA is one of the technologies enabled by NVIDIA in the GPUDirect technology family, allowing network cards to bypass CPU memory replicas and operating system routines and directly send or receive data accessing GPU memory. GPUDirect RDMA can be enabled by any network framework that uses Ethernet, InfiniBand, or RoCE。

RDMA GPU data path with GPUNetIO

DMA provides direct access between the main memory of two hosts without involving the operating system, cache, or storage. This enables data transmission to have high throughput, low latency, and low CPU utilization. This is achieved by registering and sharing a local memory area so that remote hosts know how to access it.

Applications that require two peers to exchange data through RDMA typically follow three basic steps:

Step 1- Local Configuration: Each peer creates RDMA queues and memory buffers locally to share these resources with other peers.

Step 2- Exchange Information: Use out of band (OOB) mechanisms (such as Linux sockets) to exchange information about RDMA queues and memory buffers to be accessed remotely between peers.

Step 3- Data Path: Two peers use remote memory addresses to perform RDMA reads, writes, sends, and receives to exchange data.

The DOCA RDMA library implements RDMA communication through InfiniBand or RoCE according to the three steps listed above, all of which are executed by the CPU. With the introduction of the new GPUNetIO RDMA feature, applications can perform step 3 on the GPU, using the CUDA kernel to manage the data path of RDMA applications, while steps 1 and 2 remain unchanged as they are independent of the GPU data path.

The benefits of moving the RDMA data path to the GPU are the same as those in the Ethernet use case. In network applications where data processing occurs on GPUs, network communication is offloaded from the CPU to the GPU, enabling it to become the main controller of the application, eliminating additional latency required for interaction with the CPU, knowing when data is ready and where it is located, which also releases CPU cycles. In addition, GPUs can simultaneously manage multiple RDMA queues in parallel, for example, each CUDA block can publish RDMA operations on different RDMA queues。

IB Verbs and DOCA GPUNetIO Performance Testing

In DOCA 2.7, a new DOCA GPUNetIO RDMA client server code example was introduced to demonstrate the usage of the new API and evaluate its correctness. This article analyzes the performance comparison between GPUNetIO RDMA function and IB Verbs RDMA function, reproducing a micro benchmark from the well-known perftest suite.

In short, perftest is a set of micro benchmarks used to measure RDMA bandwidth (BW) and latency between two peers (server and client) using basic RDMA operations. Although the network control part occurs in the CPU, it can be specified whether data resides in GPU memory by enabling GPUDirect RDMA and specifying the — use_cuda flag.

Generally speaking, the RDMA write unidirectional BW benchmark (i.e. ibw_rite_bw) publishes a list of write requests for messages of the same size on each RDMA queue, with a fixed number of iterations, and commands the NIC to perform the published write operation, which is called the “doorbell ringing” program. To ensure that all writes have been issued, it polls the completion queue before entering the next iteration, waiting for confirmation that each write has been executed correctly. Then, for each message size, the total time spent on publishing and polling can be retrieved, and BW can be calculated in MB/s.

In the main loop iteration of Ibw_rite_bw performance testing, the CPU issues an RDMA write request list, commands the NIC to execute these requests, and then waits for completion before moving to the next iteration. After enabling the CUDA flag, the data packets to be written will be retrieved locally from GPU memory instead of CPU memory.

The experiment involves copying the ibw_rite_bw micro benchmark using the DOCA library, establishing a client server connection using DOCA RDMA as the control path on the CPU, and publishing writes within the CUDA kernel using DOCA GPUNetIO RDMA as the data path. This comparison is not entirely consistent, as perftest uses GPUDirect RDMA to transmit data, but network communication is controlled by the CPU, while DOCA GPUNetIO uses both GPUDirect RDMA and GPUDirect Asynchronous to control network communication and data transmission from the GPU. The goal is to demonstrate that DOCA GPUNetIO RDMA performance is comparable to IB Verbs performance testing, which is considered the baseline.

In order to reproduce the ibw_rite_bw data path and measure the time taken to issue RDMA write operations for each message size, the CPU records a CUDA event, starts the rdmaw_rite_bw CUDA kernel, and then records the second CUDA event. This should provide a good approximation of the time (in milliseconds) it takes for the CUDA kernel to publish RDMA writes using the DOCA GPUNetIO function. At each iteration, the GPU CUDA kernel releases a list of RDMA write requests in parallel, with one CUDA thread per CUDA block. After synchronizing all CUDA threads, only thread 0 commands the NIC to perform a write and wait for completion, then refreshes the queue, and finally proceeds to the next iteration.