Large Language Models (LLMs) have taken the world by storm, dazzling us with their ability to write, summarize, code, and converse. But like any powerful tool, they have their limitations. What if you need your AI to draw upon your company’s latest internal documents, stay updated with real-time information, or provide answers grounded in verifiable facts rather than just its pre-trained knowledge?


This is where Retrieval Augmented Generation (RAG) steps in. RAG is a powerful technique that supercharges LLMs by connecting them to external knowledge sources, leading to more accurate, timely, and trustworthy AI systems.


This post is the start of a series on the topic and will cover the very basic setting of RAG, focusing on why it’s essential and how it functions at a fundamental level. In the next posts of this series, we’ll dive deeper into a step-by-step guide for building and refining your own RAG system.

Section 1: The Quest for Informed AI – Why Do We Need RAG?

While LLMs are impressive, they aren’t oracles. Their common limitations include:

  • Knowledge Cut-off: LLMs are trained on vast datasets, but this training has a cut-off date. An LLM trained until early 2023 won’t know about significant events, research, or product updates that occurred in 2024 or 2025.
  • Potential for Hallucination: When faced with questions outside their training data or in highly specialized domains, LLMs can sometimes “hallucinate” – generating plausible-sounding but incorrect or nonsensical information.
  • Lack of Domain-Specificity (Without Costly Fine-Tuning): General-purpose LLMs might not possess the deep, nuanced knowledge required for specific industries, proprietary company data, or niche academic fields. Constantly fine-tuning massive models for every new piece of information is often impractical.
  • Opacity & Lack of Source Attribution: It’s frequently difficult to determine why an LLM provided a particular answer or from which part of its vast training data the information was derived. This “black box” nature can be problematic for applications requiring verifiability.

RAG offers a compelling solution. It allows LLMs to access and utilize external, up-to-date information at the moment they need to generate an answer. This grounds the LLM’s responses in specific, relevant data, making them more reliable and current.

Section 2: Unlocking Meaning – What are Embeddings and Why Do They Work for Retrieval?

To understand how RAG finds the “right” information, we first need to grasp the concept of embeddings.

  • What are Embeddings? The Language of Meaning for Machines.
    Raw text is challenging for computers to “understand” semantically. Embeddings bridge this gap. They are numerical representations (vectors – essentially, lists of numbers) of pieces of text (like words, sentences, paragraphs, or even whole documents). You can think of these vectors as coordinates that place each piece of text within a high-dimensional “meaning space.”
  • Why are Embeddings Meaningful? Capturing Semantic Relationships.
    The power of modern embedding models (which are sophisticated neural networks themselves) lies in their ability to create embeddings where texts with similar meanings are located “close” to each other in this vector space. Conversely, texts with dissimilar meanings are positioned further apart. For instance, in this meaning space, the embedding for “summer vacation” might be close to “holiday trip” and “beach getaway,” but distant from “quarterly earnings report” or “software installation guide.” This goes beyond simple keyword matching; these embeddings can capture nuanced semantic relationships. For example, the vector difference between the embedding for “woman” and “queen” might be very similar to the vector difference between “man” and “king,” indicating that the model has learned the concept of royalty in relation to gender (i.e., queen is to woman as king is to man).
  • Why Can Embeddings Detect Relevant Context for a Query? The Power of Proximity.
    Imagine you have a vast library of documents relevant to your needs. Using an embedding model, you can convert every meaningful chunk of text within those documents into an embedding. Now, when a user asks a question (a “query”), you can transform this query into an embedding using the exact same embedding model.
    The core principle is this: If a chunk of text from your library is semantically relevant to the user’s query, its embedding will be “close” to the query’s embedding in this shared meaning space. By identifying the chunk embeddings that are closest to the query embedding, we can retrieve the most promising pieces of background context to help answer the question.

Section 3: The Basic RAG Architecture – A Simple Flow

At its heart, RAG is a process that:

  1. Retrieves relevant information snippets from an external knowledge source.
  2. Augments the user’s original query with these retrieved snippets.
  3. Passes this augmented information to an LLM for Generation of a comprehensive answer.

Let’s break down the essential steps for a very simple, foundational RAG setup:


1. Preparing Your Knowledge Base (The “Retrieval” Groundwork):


This is the preparatory phase where you set up the information your RAG system will draw upon.

  • Get Data: It all starts with your data. For now, we’re primarily considering text-based information, which is the collection of documents, articles, databases, or any textual content you want your RAG system to be knowledgeable about. This could be internal company wikis, product manuals, customer support logs, or publicly available research papers. Eventually, we will also explore incorporating other data types like diagrams and images.
  • Decide on Chunks (Chunking): Large documents are often too unwieldy to be processed efficiently by LLMs or to allow for precise information retrieval. Therefore, you break these documents down into smaller, more manageable pieces called “chunks.” A chunk might be a single paragraph, a few sentences, or a section defined by a specific heading. Why do we do this? Chunking allows for more targeted retrieval (finding the most relevant small piece of info) and helps ensure that the retrieved context fits within the LLM’s input limitations (its “context window”).
  • Compute Embeddings for Chunks: Each text chunk is then fed into an embedding model, which outputs a numerical vector (the embedding) for that chunk. This process is repeated for all chunks, creating a searchable “vector index” of your entire knowledge base. Each chunk is now represented by its unique meaning vector in that high-dimensional space we discussed.

2. Answering a User’s Query (The “Augmentation & Generation” Runtime):


This is what happens when a user interacts with your RAG system.

  • Embed the User’s Query: When a user asks a question (e.g., “What were our Q1 sales figures for Product X?”), their query text is converted into an embedding using the exact same embedding model that was used to create the chunk embeddings. This is crucial because it ensures both the query and the chunks are represented in the same “meaning space,” allowing for meaningful comparisons.
  • Retrieval (Finding Relevant Chunks): The system now compares the user’s query embedding against all the chunk embeddings stored in your vector index.
    • Using Cosine Similarity: A very common and effective method for measuring the “closeness” or “similarity” between two embedding vectors is cosine similarity. It calculates the cosine of the angle between two vectors. If the vectors point in nearly the same direction (angle close to 0 degrees), the cosine similarity is close to 1, indicating high semantic similarity. If they are orthogonal (angle of 90 degrees), the similarity is 0 (unrelated). If they point in opposite directions, it’s -1 (dissimilar).
    • The system identifies the top ‘N’ chunks (e.g., the top 3 or 5) whose embeddings have the highest cosine similarity scores with the query embedding. These are deemed the most relevant pieces of context from your knowledge base.
  • Answer Generation (The LLM’s Role):
    • The actual text content of these top ‘N’ retrieved chunks is then combined with the original user query. This often involves creating a new, “augmented prompt” that might look something like:
      Context:  
  \[Text from retrieved chunk 1\]  
  \[Text from retrieved chunk 2\]  
  \[Text from retrieved chunk 3\]

  Question: \[Original user query\]

  Based on the context provided, please answer the question.
    • This augmented prompt is then sent to a Large Language Model (LLM). The LLM uses the provided context snippets to formulate a relevant and informed answer to the user’s original question.

While these steps cover the basics of RAG, the true difficulty often lies in thoroughly evaluating the quality of the entire system and iteratively improving upon this foundational setting, topics we will explore in detail in the next parts of this series.


Disclaimer: This post is the result of me chatting with an AI to structure my thoughts on this topic. The AI then kindly drafted this summary based on our talk and my outline, serving as my personal ‘don’t forget!’ note for the future – because apparently, my brain isn’t a perfect recording device. I’ve made minor edits for clarity.