Evaluating and Enhancing RAG Pipeline Performance Using Synthetic Data 

As large language models (LLM) gain popularity in various question-answering systems, retrieval-augmented generation (RAG) pipelines have also become a focal point. RAG pipelines combine the generation power of LLMs with external data sources and retrieval mechanisms, enabling models to access domain-specific information that may not have existed during fine-tuning.

RAG performance heavily depends on the quality of the underlying text embedding models. Poor embedding models lead to subpar retrieval results, which degrades the overall performance of the RAG system. This is especially true when working with internal, enterprise-specific data, where out-of-the-box models may not deliver the desired accuracy.

To ensure that a RAG system works effectively with your specific data, you must evaluate whether pretrained embedding models perform well on your corpus. This evaluation requires annotated data that closely resembles the types of queries that end users will submit. For example, a sample annotated dataset might include the following:

Question: How do I reset my enterprise VPN password?Passage: To reset your VPN password, visit the IT help portal, select ‘Reset Password,’ and follow the on-screen instructions.Answer: Visit the IT help portal and select 'Reset Password'

In many cases, off-the-shelf embedding models don’t yield optimal results when applied to domain-specific data. This creates a need to customize these models using annotated data that shares the same distribution as the enterprise corpus.

In this post, we discuss how to create synthetic data to evaluate and customize your embedding models using the new NVIDIA synthetic data generation (SDG) pipelines. By generating high-quality synthetic annotations, you can enhance the performance of your embedding models and optimize retrieval accuracy for your enterprise-specific use cases.

Customizing and evaluating embedding models 

Embedding models play a critical role in RAG systems by converting both the document corpus and user queries into dense numerical vectors. These embeddings are stored in a database, enabling the RAG system to efficiently retrieve documents most relevant to a given query. 

While pretrained embedding models work well for general text, they often fail to capture the nuances of domain-specific data. This gap leads to unreliable search results, missed connections, and poor RAG performance.

To address this issue, embedding models must be evaluated and customized using domain-specific data. This customization adjusts the model’s internal weights, enabling it to better understand the contextual meanings relevant to the target enterprise application.

For example, consider the word “pitch“ in different contexts:

• Sports: “The baseball pitch was perfect.” Here, “pitch” refers to a throw.
• Business: “The startup delivered a great pitch to investors.” In this case, “pitch” means a presentation.

If an embedding model is trained primarily on sports data, the vector for “pitch” leans toward the sports context. This could lead to inaccurate retrievals when used for business queries. 

By fine-tuning the model on business-specific text, the embeddings for “pitch” shift to represent its business-related meaning, improving the accuracy of document retrievals in that domain.

Creating evaluation and customization data for embedding models is challenging

Publicly available datasets often lack relevance when applied to enterprise-specific data. These datasets are typically collected from publicly-available sources, where the vocabulary, context, and structure may not match the domain-specific content found within an enterprise. 

As a result, using public datasets for evaluating or customizing a RAG system can lead to inaccurate performance assessments and suboptimal model fine-tuning.

Creating human-annotated enterprise-specific datasets is both expensive and time-consuming. This process requires domain experts to label large volumes of data, ensuring that annotations capture the specific terminology and context of the industry. Moreover, this approach is difficult to scale, especially as the enterprises’ needs evolve over time.

Generating high-quality synthetic data with NVIDIA NeMo Curator

NVIDIA NeMo Curator improves generative AI model accuracy by processing text, image, and video data at scale for training and customization. It also provides prebuilt pipelines for generating synthetic data to customize and evaluate embedding models. 

The SDG pipeline for generating RAG evaluation data is specifically designed to generate high-quality question-answer (QA) pairs that assess how well an embedding model performs. It consists of three key components, each ensuring that the generated data is contextually relevant, appropriately challenging, and grounded in the source material:

• QA pair-generating LLM
• Embedding model-as-a-judge for question easiness
• Answerability filter for grounding

NVIDIA NeMo Retriever, a key component of the NVIDIA NeMo platform, also enhances RAG applications with faster, more precise, multimodal data extraction and retrieval with optimized models packaged as NVIDIA NIM microservices. After you customize your embedding models for your domain, deploy them with NeMo Retriever to evaluate and scale in the production pipeline. 

QA pair-generating LLM

This component uses any NVIDIA NIM LLM to generate QA pairs from seed documents. We provided optimized system prompts that guide the LLM to create more context-aware and relevant questions.

Example Input:Seed Document: "Our enterprise offers cloud-based cybersecurity solutions to prevent data breaches."Example Output:Question: "What type of solutions does the enterprise offer?"Answer: "Cloud-based cybersecurity solutions."

By using system prompts designed specifically for this task, the LLM produces questions that align with the context of the seed data, ensuring higher quality outputs.

Embedding model-as-a-judge for question easiness

To ensure that the generated questions cover a range of difficulties, this component uses an embedding model to evaluate and rank the complexity of each question. 

The embedding model is used as a judge to filter out generated questions based on their cosine similarity with context documents. The thresholds are determined by calibrating with manually annotated datasets to mimic human judgment, and these thresholds can be easily customized to suit specific needs.

The difficulty distribution of the dataset can be controlled by changing these thresholds. Having lower thresholds can result in a higher percentage of harder questions and can help generate harder evaluation datasets. 

By applying these thresholds, the pipeline ensures that the final dataset includes a balanced mix of easy, medium, and difficult questions, improving the robustness of model evaluations. 

Answerability filter for grounding

The final component acts as a filter to ensure that each generated question is directly grounded in the seed document. This step prevents irrelevant or hallucinated questions from being included in the dataset, maintaining the quality and reliability of the evaluation data. This ensures that only the directly supported by the source material are retained, enhancing the credibility and relevance of the final dataset.

NeMo Curator also provides a pipeline for generating customization data for embedding models. Apart from the components in the evaluation pipeline, it adds a hard negative mining module.

Understanding hard-negative mining

Hard negatives play a crucial role in enhancing the performance of contrastive learning for embedding models. These samples are defined as those that are difficult to distinguish from positive samples but contain contexts that don’t contain answers to the query. 

By incorporating hard negatives, models are forced to learn more discriminative features, improving their ability to differentiate between similar yet distinct data points. This approach helps fine-tune the model’s decision boundary, leading to more robust and generalizable representations. 

During customization, the goal is to increase the distance between embeddings of negative documents and those of the question while decreasing the distance between positive documents and the question. This process helps the model learn to better differentiate between relevant and irrelevant information.

For most embedding models, the distance between a question and irrelevant documents is already high. As a result, simply using random negative documents doesn’t significantly impact the model’s learning. 

Instead, the model is trained using hard negatives—negative documents that are more challenging to distinguish because their embeddings have a higher cosine similarity to the question. By focusing on these harder cases, the model is pushed to refine its understanding of subtle differences, leading to more accurate retrieval results.

You can use hard negative documents using the following methods:

• Top-K selection: The system identifies the top K negative documents that have the highest cosine similarity to the question. By selecting such negatives, the model is trained to better separate relevant and irrelevant documents that are close in vector space. This is one of the simplest methods.  
• Threshold-based selection: An alternative approach is to set minimum and maximum thresholds for cosine similarity between negatives and the question and select the top K negative documents that lie within these thresholds. This ensures that only the most challenging negatives are used during training, further fine-tuning the model’s decision boundary. 
• Positive-aware mining: The method uses the positive relevance score as an anchor to more effectively remove false negatives. This approach is designed to address the challenge of selecting high-quality hard negative passages for contrastive learning. We define the maximum negative similarity threshold as a percentage of the positive score. This method performs very well, with the threshold set to 95% of the positive score. For more information about the ablation study, see NV-Retriever: Improving text embedding models with effective hard-negative mining.

With the hard-negative mining, you can generate high quality data to customize your embedding models and increase the accuracy of your RAG applications.

Summary

In this post, we discussed how the SDG pipelines from NeMo Curator simplify generating high-quality datasets, enabling the precise evaluation and customization of text embedding models.

With these enhanced datasets, you can effectively evaluate and fine-tune RAG performance, gaining insights into how well your retriever systems perform and identifying ways to improve accuracy and relevance. 

For more information about how to get started, see the following resources:

• NeMo Retriever Synthetic Data Generation tutorial
• Transform an Enterprise Data Platform with Generative AI and RAG [S72205] GTC session

With NeMo Retriever embedding microservices, you can further optimize multimodal document retrieval, improving the overall accuracy of RAG applications at scale with significantly lower costs.