Welcome back to our deep dive into the world of Retrieval Augmented Generation. In our first post, “Grounding LLMs: An Introduction to Retrieval Augmented Generation (RAG)”, we established the fundamental “Why” of RAG—its power to combat hallucination and ground LLMs in factual, up-to-date information. In our second post, “RAG Unpacked: A Deep Dive into Core Components and Customization Strategies”, we explored the “How,” dissecting the many knobs and levers we can tune, from chunking strategies and embedding models to complex retrieval algorithms.


Now, we arrive at the most critical question: With all these configurable parts, how do we quantitatively know if our choices are effective? How do we prove that one RAG pipeline is better than another? For this we need a robust evaluation framework. The central theme of this post is that evaluating a RAG system is fundamentally a two-part problem, mirroring its architecture:

  1. Retrieval Evaluation: Did we find the right information?
  2. Generation Evaluation: Did we use that information correctly to form a good answer?

Getting retrieval right is the biggest lever for success, so we’ll start there, building a rigorous methodology from the ground up before assessing the final generated answer.

Section 1: Evaluating the Foundation – The Retrieval System

An LLM, no matter how powerful, cannot produce a correct answer from incorrect or irrelevant context. The quality of your retrieval system is the ceiling for the quality of your entire RAG application. This is where we must focus the majority of our evaluation efforts.

Pre-flight Check: Are Your Embeddings Discriminating?

Before you even start building complex evaluation pipelines, it pays to perform a simple sanity check on your embeddings. The risk, especially when working with highly specialized, domain-specific documents (e.g., medical research papers on diabetes), is the “semantic blob”. This occurs when a generic embedding model, not fine-tuned for your domain, sees all your text as broadly similar, mapping every chunk into the same dense region of the vector space. If your embeddings can’t tell the difference between distinct concepts, your retrieval system is doomed before it starts.

Here’s how to inspect this:

  • Technique 1: Pairwise Similarity Distribution: Take a random sample of your text chunks (say, 1000 chunks) and calculate the cosine similarity for all pairs. Plot a histogram of these similarities. If the distribution is heavily skewed towards 0.9 or higher, it’s a massive red flag. It means your model sees everything as a near-duplicate.
  • Technique 2: Visualization: Use a dimensionality reduction technique like t-SNE or UMAP to create a 2D visualization of your chunk embeddings. Do semantically distinct concepts (which you might know from your domain expertise) form visible clusters? Or do you see one large, undifferentiated cloud? A healthy embedding space will show distinct clusters and separation.

Performing this check early can save you hours by validating your choice of embedding model from the outset.

Building Your Workbench: Creating an Evaluation Dataset

To measure retrieval performance, we need a ground truth. The core unit of this ground truth is the (Question, Positive_Chunk) pair. Manually creating thousands of these pairs is tedious and unscalable. A far more pragmatic approach is to use a powerful LLM to create a synthetic dataset for you.

  1. Iterate: Go through each of the text chunks in your document store.
  2. Generate: For each chunk, use a state-of-the-art LLM with a precise prompt: “Based only on the information in the following text, generate one high-quality, specific question that can be answered by it. The question should not require any outside knowledge.”
  3. Store: Save the generated question and the ID of the source chunk.

A note on synthetic data quality: Be aware that this method can sometimes produce questions that are “too easy,” as they may directly mirror the phrasing of the source text, inflating retrieval scores. It is an excellent starting point, but complementing it with a smaller, manually curated set of more challenging questions is a robust practice.


A note on statistical significance: Your evaluation dataset must be large and representative enough to draw meaningful conclusions. Calculating metrics on just 20 questions can be misleading. Aim for a set of at least a few hundred high-quality data points to ensure your results are statistically significant.

The “In-Batch Negative” Problem & Duplicate Chunks

With a set of positive pairs, a tempting first step is to assume that for any given question, its paired chunk is “positive” and every other chunk in your entire corpus is “negative.” This is known as using “in-batch negatives.” The core logical flaw here is conflating “not the chosen positive chunk” with “is definitively a negative chunk.”


This assumption is flawed, because you might have near-duplicate chunks. Imagine you have a v1 and a v2 of a policy document. They might be 99% identical. If your dataset links a question to a chunk in v1, a retrieval system that correctly finds the equivalent chunk in v2 would be marked as a failure. This unfairly penalizes the system for finding semantically identical information.


Before building metrics, inspect your corpus for these duplicates by finding pairs of chunks with near-perfect (e.g., >0.98) cosine similarity.

Level Up: Generating Explicit Negative & Hard Negative Pairs

To build a truly robust evaluation set, we need to move beyond simple positive pairs. We can once again use an LLM as a judge to explicitly label chunks as “negative” or even identify other “positive” chunks.

  1. Retrieve: For a question from your synthetic dataset, run it through your retrieval system and get the top-k results (e.g., top 10).
  2. Judge: For each retrieved (question, chunk) pair that isn’t your original ground-truth positive, ask a powerful LLM: “Does the following text contain the information needed to answer this question? Answer with only ‘Yes’ or ‘No’.”
  3. Enrich: This process enriches your dataset in three ways:
    • It validates new positive pairs (when the LLM says “Yes” for a different chunk).
    • It creates explicitly negative pairs (when the LLM says “No”).
    • It helps you find “hard negatives”—chunks that are topically similar but factually incorrect. These are the most valuable negatives, as they truly test your system’s precision. For example, a question about “Lisa A., the CFO” might incorrectly retrieve a chunk about “Lisa E., the CEO.”

The Metrics That Matter for Retrieval

Now that you have a high-quality, labeled dataset of positive and negative pairs, you can finally calculate meaningful metrics.

  • Hit Rate: The simplest metric. Did you retrieve at least one correct chunk in your top-k results? It’s a binary (Yes/No) measure, useful for a quick pulse check.
  • Mean Reciprocal Rank (MRR): This metric evaluates how high up in the ranking the first relevant document appears. It’s perfect for fact-based Q\&A where finding a single correct answer quickly is what matters. The score ranges from 0 to 1, where 1 is a perfect score.MRR=∣Q∣1​i=1∑∣Q∣​ranki​1​
    Here, ∣Q∣ is the total number of questions, and ranki​ is the rank of the first correct chunk for the i-th question. If the correct chunk is not found within the retrieved results for a given question (e.g., not in the top 10), its reciprocal rank is 0, effectively penalizing the system for the miss.
  • Precision@k & Recall@k: These metrics are crucial when a question could be answered by multiple chunks.
    • Precision@k: Of the top-k retrieved chunks, what fraction is relevant? This answers: “How much junk is in my search results?”
    • Recall@k: Of all the known relevant chunks in your dataset, what fraction did you find in the top-k? This answers: “How much important information is my system failing to find?”

A significant challenge arises when your retrieval results contain unlabeled data—chunks for which you have neither a positive nor a negative label. One pragmatic approach is to simply ignore the unlabeled items for the purpose of the metric. The evaluation is then performed only on the subset of retrieved chunks with an explicit label. For example, if your system retrieves 10 chunks and among them are 2 known positives, 4 known negatives, and 4 unlabeled chunks, your Precision@10 would be calculated based only on the 6 items with a known label, resulting in a score of 2/6.

Section 2: Evaluating the End Product – The Generated Answer

Even with perfect retrieval, the LLM can still fail by misinterpreting the context, ignoring it, or “hallucinating” by adding information that wasn’t provided. This is the second pillar of our evaluation. Here, we can use a powerful LLM as a judge, following two main approaches depending on whether we have a labeled dataset of ground-truth answers.

Approach 1: Evaluation without Ground-Truth Answers

When you don’t have a pre-labeled dataset of correct answers, you can still assess the quality of the generated answer by comparing it directly against the retrieved context. This approach focuses on the internal consistency and quality of the output, based on the (Question, Context, Answer) triplet.

The key dimensions to evaluate here are:

  • Faithfulness: This dimension evaluates whether the answer is factually consistent with the retrieved source documents. An answer is considered unfaithful if it distorts information or directly contradicts the source material. It’s the primary guard against hallucination.
  • Groundedness: This evaluates whether every part of the answer is explicitly supported by the retrieved content. Even if an answer is factually correct (faithful to general world knowledge), it is considered ungrounded if the supporting information wasn’t part of the retrieved evidence. This ensures the RAG system is truly “closed-book” and relies only on the documents you provided.
  • Answer Relevancy: This assesses whether the answer directly addresses the user’s question. A response may be truthful and well-grounded but still fail if it doesn’t respond to the actual query or provides off-topic information.
  • Conciseness: This measures whether the answer is succinct and free of unnecessary information. A concise answer delivers the needed information without digressions, repetition, or verbose phrasing, improving readability and user satisfaction.

These dimensions are complementary. For example, an answer can be faithful (factually correct in the world) but ungrounded (not supported by the context). A high-quality answer aims to meet all criteria. To automate this, you can use an LLM as a judge.

  • Prompt Example (for Faithfulness): “Analyze the provided context and answer. For each sentence in the answer, determine if it is directly supported by the information in the context. Conclude with a single word: ‘Faithful’ or ‘Unfaithful’.”

Approach 2: Evaluation with Ground-Truth Answers

For the most robust, end-to-end evaluation of correctness, you need a labeled dataset containing ideal question-answer pairs: (Question, Ground_Truth_Answer). This dataset can be created manually by experts or synthetically, again using a powerful LLM to generate answers for a set of questions.


With this dataset, the LLM-judge’s role changes. Instead of checking against the context, it compares the RAG system’s generated answer directly against the known-good, ground-truth answer.

  • Prompt Example: “Given the question, the ground-truth answer, and the generated answer, assess if the generated answer is semantically equivalent and factually consistent with the ground-truth answer. Conclude with ‘Correct’ or ‘Incorrect’.”

Thankfully, you don’t have to build all this from scratch. Open-source frameworks like RAGAs have encapsulated both of these evaluation approaches into pre-built metrics.

Section 3: Advanced Considerations & Practical Hurdles

Adversarial Testing & Noise Robustness

Go beyond your synthetically generated dataset and create a small, curated set of “adversarial” questions. This is a form of Noise Robustness or “Distractor Test” designed to target known ambiguities in your knowledge base (like the “Lisa A.” vs. “Lisa E.” problem) and stress-test your system’s ability to differentiate nuances.


You can also make this a direct test for both your retriever and your LLM. Manually construct a challenging scenario by putting together a set of similar and misleading chunks. Then, observe the full pipeline:

  • Test the Retriever: Does your retrieval mechanism pull in the full set of similar and misleading chunks as expected? Or is it able to omit the truly misleading ones in the first place? Both outcomes provide valuable insight.
  • Test the LLM: Given a context that includes the correct chunk alongside several misleading ones, can the LLM correctly synthesize an answer? This might give you a strong indicator of whether you need to use a more powerful model, especially if you aren’t already using a state-of-the-art/flagship model.

The Multi-Chunk Reasoning Challenge

A major real-world pitfall is that complex questions often require synthesizing information from multiple chunks. Your evaluation must account for this. The core unit of your ground truth becomes (Question, [chunk_id_1, chunk_id_2, …]).


This requires more sophisticated metrics. Instead of a simple MRR, you need to measure Context Recall—the proportion of necessary ground-truth chunks that were successfully retrieved in the top-k results. A high Context Recall score indicates your retriever is gathering all the necessary pieces of the puzzle for the LLM to work with.

The Fragility of Evaluation: The Chunking Conundrum

A critical point to remember: your retrieval evaluation dataset is intrinsically tied to your chunking strategy. If you build a dataset based on 256-token chunks and later decide to re-index your content using 512-token chunks, your entire (question, positive_chunk_id) mapping becomes invalid. Be aware that changing fundamental preprocessing steps will likely require you to regenerate your evaluation set.

The Practicalities of Evaluation: Cost and Latency

Evaluation, especially using an LLM-as-a-judge, has a real cost (API calls) and latency. Running an evaluation on 10,000 question-answer pairs can be slow and expensive. This practical constraint means you should use an evaluation “funnel”: start with cheap, fast metrics like MRR to iterate quickly on your retriever, and then use the more expensive end-to-end, LLM-judged evaluations on a smaller, curated set of examples to validate the final pipeline.

Closing the Loop: Leveraging User Feedback for Continuous Improvement

This is a crucial concept for any production-level RAG system. It closes the loop and moves evaluation from a static, offline process to a dynamic, continuous improvement cycle. User feedback is the ultimate ground truth. While automated metrics are great proxies, nothing tells you if your system is actually useful better than the users themselves. This feedback can then be used to create a high-quality dataset for fine-tuning and further evaluation.

  • Types of Feedback:
    • Explicit Feedback: The most direct method. Add thumbs up/down buttons, star ratings, or a simple “Was this answer helpful?” form next to each generated answer.
    • Implicit Feedback: More subtle but also powerful signals. Track user behaviors like copying the generated answer (strong positive signal), immediately rephrasing the question (strong negative signal), or clicking on cited source documents (indicates engagement).
  • How to Use the Feedback:
    • For Evaluation: The aggregated feedback (e.g., percentage of “thumbs up” responses) becomes a business-level KPI for the RAG system’s performance. Furthermore, the collected question/answer pairs that receive positive feedback can be added to your “golden” evaluation dataset, continuously improving its quality and coverage.
    • For Optimization: The collected (Question, Answer, Context, Feedback) tuples are a goldmine. It’s crucial to log the retrieved chunks (the Context) that led to each answer. This data can be used to create a high-quality dataset for fine-tuning. A “thumbs up” provides a strong positive signal that the retrieved chunks were relevant to the question. Conversely, a “thumbs down” can serve as an indirect negative signal, suggesting the retrieved context may have been poor, helping you identify areas where your retrieval step needs improvement.

A Note on Rerankers

If you use a reranker (like a Cross-Encoder) to improve the precision of your top results, the evaluation process remains the same. You simply apply the metrics (MRR, Precision@k, Context Recall) to the final, reranked list of chunks to measure the “lift” or improvement provided by the reranking step.

Conclusion: A Blueprint for Building a High-Quality RAG System

We’ve moved from the art of building a RAG pipeline to the science of measuring it. Here is a step-by-step blueprint to guide your process from start to finish:


Phase 1: Foundation & Setup

  • Data Preparation & Parsing: Start by parsing and cleaning your source documents (PDFs, HTML, etc.) to extract clean, relevant text.
  • Strategic Chunking: Choose a chunking strategy that preserves semantic context.
  • Establish a Strong Baseline: Begin with a managed AI Search service that provides powerful features like hybrid search and reranking out-of-the-box. This gives you a high-quality first version from which to iterate.
  • Embedding Model & Data Validation: Select an embedding model and perform a “pre-flight check” to ensure it can discriminate between concepts in your domain (avoid the “semantic blob”). Simultaneously, check whether duplicates are a problem in your dataset, as this will heavily impact evaluation down the line.

Phase 2: Core Evaluation Loop

  • Build a Golden Dataset: Synthetically generate a high-quality (Question, Positive_Chunk, [optional] Ground_Truth_Answer) evaluation set.
  • Evaluate the Retriever: Use metrics like MRR and Context Recall to test if you’re finding the right information. Iterate on your retrieval strategy (e.g., trying different search algorithms or rerankers) until these metrics are strong.
  • Evaluate the Generator: Use an LLM-as-a-judge to measure Faithfulness, Groundedness, and Relevancy. Iterate on your prompting strategy and model choice until the answers are high-quality.

Phase 3: Hardening & Stress-Testing

  • Adversarial Testing: Create manual “distractor tests” to identify specific failure modes and challenge your system with nuance and ambiguity.

Phase 4: Deployment & Continuous Improvement

  • Implement Feedback Mechanisms: Deploy your system with explicit (thumbs up/down) and implicit (user actions) feedback loops.
  • Monitor and Iterate: Use live user feedback as your ultimate evaluation metric and to continuously create new data for fine-tuning and improving your system over time.

We’ve covered the what, the why, and the how of Retrieval Augmented Generation. This final post on evaluation provides the most critical tool in your arsenal: the ability to prove your system works. Building a RAG pipeline is a significant step, but building one with a data-driven, systematic evaluation process is what separates a good prototype from a great, production-ready application. Armed with these strategies, you can now iterate with confidence, tune your system’s many knobs with precision, and ultimately deliver solutions that are both powerful and trustworthy.


Disclaimer: This post is the result of me chatting with an AI to structure my thoughts on this topic. The AI then kindly drafted this summary based on our talk and my outline, serving as my personal ‘don’t forget!’ note for the future – because apparently, my brain isn’t a perfect recording device. I’ve made minor edits for clarity.