Leveraging accelerated Ethernet networking and network storage to expand enterprise RAG (Robotics, Automation, and AI).

Leveraging accelerated Ethernet networking and network storage to expand enterprise RAG (Robotics, Automation, and AI).

In the realm of generative AI, machines can not only learn from data but also generate human-like text, images, videos, and more. Retrieval-Augmented Generation (RAG) is a groundbreaking approach in this field.

The RAG workflow is based on Large Language Models (LLMs), which can understand queries and generate responses. However, LLMs have limitations, including training complexity and a lack of up-to-date (and sometimes proprietary) information. Additionally, when not trained on specific data to answer prompts, they tend to produce hallucinations and synthesize factually incorrect information. RAG enhances queries by providing LLMs with enterprise-specific information, helping to overcome these limitations.

In this article, we discuss how RAG helps enterprises create high-quality, relevant, and engaging content for various business use cases. We delve into the technical challenges of scaling RAG to handle vast amounts of data and user demands, and how these challenges can be addressed using a scalable architecture supported by NVIDIA GPU computing, accelerated Ethernet networking, network storage, and AI software.

RAG enables enterprises to fully leverage their data

A typical RAG workflow utilizes vector databases, specialized data management systems tailored for performing similarity searches, to store and retrieve enterprise-specific information relevant to queries.

By integrating RAG into their information systems, enterprises can leverage vast amounts of internal and external data to generate insightful, up-to-date, contextually relevant content. This integration represents a significant leap forward, enabling companies to harness their data and domain expertise to unlock new avenues for personalized customer interactions, streamline content creation, and enhance the efficiency of knowledge use cases.

However, deploying RAG at an enterprise scale comes with a series of challenges, including the complexity of managing hundreds of datasets and thousands of users. This necessitates a distributed architecture capable of efficiently handling the processing and storage demands of such large-scale operations.

To scale this architecture, you must embed, vectorize millions of documents, images, audio files, and videos, and index them, while also accommodating newly created content daily.

Another challenge is ensuring low-latency responses for interactive multimodal applications. Integrating data from enterprise applications and both structured and unstructured data stores requires real-time processing and responsiveness, which can be challenging to achieve at scale.

Data indexing and storage for generative AI also present challenges.

While traditional enterprise applications can compress data and store it for efficient retrieval to support indexing and semantic search, RAG-based databases can expand to be more than ten times larger than the original text files and their associated metadata. This leads to significant challenges in terms of data growth and storage.

To achieve optimal results, enterprises must invest in accelerated computing, networking, and storage infrastructure, which is crucial for handling the vast amounts of data required to train and deploy RAG models.

Scalable and efficient RAG inference

At GTC 2024, NVIDIA introduced a generative AI microservices catalog, providing developers with enterprise-grade foundational modules for creating and deploying custom AI applications.

Enterprises can use these microservices as the foundation for creating RAG-driven applications. By combining them with NVIDIA RAG workflow examples, you can accelerate the building and productization process of generative AI applications.

In this article, we benchmarked these RAG workflow examples using multi-node GPU computing inference, accelerated Ethernet networking, and network-attached storage. Our test results show that high-performance networking and network-attached storage enable efficient and scalable generative AI inference, allowing enterprises to develop RAG-driven applications that can scale to thousands of users while facilitating continuous data processing.

The following diagram illustrates the RAG workflow, which includes two stages and a data pipeline

In the first stage, data extraction converts documents and other data formats into digital embeddings, which are then indexed in a vector database. This process enables efficient retrieval of relevant documents based on similarity scores.

The query stage begins when a user inputs a question, which is also converted into an embedding and used to search for related content in the vector database. After retrieving the relevant content, the system passes it to the LLM for further processing. The original input question along with the augmented context is provided to the LLM, which generates more accurate answers in response to the user’s query.

This workflow supports efficient retrieval and generation of information, making it a powerful tool suitable for various enterprise applications.

Accelerated Ethernet networking and network-attached storage excel in data extraction

We initially tested the data extraction process on a single GPU node. The following diagram shows the test setup using a DGX system equipped with 8 A100 GPUs and a network-attached all-flash storage platform designed for object storage workload.

The DGX system uses accelerated NVMe-over-Fabrics (NVMe-oF) and Amazon S3 object storage protocols via NVIDIA ConnectX-7 NICs and is connected by two NVIDIA Spectrum SN3700 switches.

We compared the embedding and indexing performance of PDF documents (including text and images) using the NeMo Retriever microservice. This comparison involved the DGX system and network-attached storage against direct-attached storage (DAS).

The results of the single-node data extraction benchmark are shown. It indicates that using network-attached storage with Amazon S3 improved data extraction speed by 36% and reduced processing time by 122 seconds compared to DAS. This suggests that network-attached storage is a better choice for data extraction, also depending on network speed and latency.

Accelerated Ethernet networking is crucial for providing robust, high-performance, and secure connections. In addition to enhancing document embedding, network-attached storage also provides various enterprise-grade data management capabilities.

Then, we tested using a multi-node RAG setup, which employs a distributed microservices architecture connected through NVIDIA BlueField-3 DPU. As multiple nodes run in parallel to upload embeddings, compute indices, and insert into the vector database, performance improves.

We compared the performance of using directly attached SSDs in each server versus network-attached storage. For the SSDs within the servers, MinIO served as the object storage layer. For network-attached storage, we bypassed MinIO and tested the storage system’s native Amazon S3 object interface.

The results show that multi-node setups provide faster performance than using a single node, reducing processing time by nearly 102 seconds. These results demonstrate the performance advantages of combining multi-node GPU acceleration with enterprise-grade network-attached storage.

Advantages of network-attached storage for RAG-driven applications

Network-attached storage supports accessing blocks, files, and objects over the network without directly connecting storage media to servers.

Network-attached storage not only provides significant performance advantages for RAG-based applications but also offers additional enterprise benefits, making it the ideal data platform for enhancing natural language processing. Here are the advantages of network-attached storage for RAG workflows:

Real-time Streaming Data Extraction: Network-attached storage supports extracting real-time streaming data from various sources (e.g., social media, the web, sensors, or IoT devices). RAG applications can use this data to generate up-to-date and relevant content. DAS might struggle with handling large volumes of fast-moving stream data or might require additional processing or buffering to store the data.

Scalability: It’s easier to scale storage capacity by adding more disks or devices without impacting performance or data availability. In contrast, DAS has limited scalability, potentially requiring downtime or reconfiguration for storage upgrades.

Metadata Annotation: Network-attached storage supports annotating data with metadata (e.g., tags, categories, keywords, or summaries). RAG applications can use this metadata to retrieve and rank data sources based on queries or context. DAS might not support data annotation or might require a separate database or index to store metadata.

Utilization: Network-attached storage (NAS) allows multiple users and applications to access the same data simultaneously, optimizing storage resource utilization without creating duplicate data or conflicts. In contrast, direct-attached storage (DAS) might lead to underutilized or overused storage, depending on demand and data distribution within specific servers.

Reliability: Network-attached storage increases reliability and data availability by protecting data from disk failures, network issues, or power outages using advanced RAID functionality or other methods. In contrast, DAS might lose or corrupt data in the event of disk or server failures because DAS cannot leverage RAID or other redundancy technologies.

Deduplication: Network storage reduces storage space and network bandwidth by eliminating duplicate or redundant data across files or devices. DAS might store multiple copies of the same data, wasting storage space and network resources.

Data Source Citations: Network-attached storage (NAS) can provide citations for data sources, such as URLs, authors, dates, or licenses. RAG applications can use this information for attribution, validation of data sources, and ensuring the quality and reliability of generated content. DAS might not provide data source citations or might require manual or external methods to track data sources.

Backup: Network-attached storage facilitates data backup and recovery by creating data copies in different locations or devices using snapshots, replication, or other methods. DAS might require manual or complex backup procedures, which can be time-consuming or error-prone.

Data Protection and Retention: Network-attached storage ensures data protection and retention by using encryption, compression, or other technologies to protect data from unauthorized access or modification. It also manages the data lifecycle using policies, rules, or regulations, such as data creation, deletion, or archiving. In contrast, DAS might not provide data protection and retention features or might require additional software or hardware to achieve data security and governance.