What is RAG? Retrieval-Augmented Generation Explained (2026)

Prerequisites & Setup

You need basic Python (functions, virtual environments, pip), the ability to make an HTTP request, and an API key for OpenAI or Azure OpenAI. That is it.

python -m venv .venv
source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install openai chromadb sentence-transformers cohere langchain numpy

Set your API keys as environment variables. Get an OpenAI key (or use Azure OpenAI in EU-region for production). Optionally, get a Cohere key for re-ranking.

Why RAG Exists (The Story)

In 2019, if you asked a language model "what is our company's refund policy?", you had three bad options:

• Fine-tune the model on your policy. Expensive ($1,000s), slow (days), and every policy update meant re-training the model.
• Paste the entire policy into every prompt. Worked for one document. Fell apart at the first thousand-page knowledge base.
• Live with hallucinations. The model invented refund policies that did not exist.

In May 2020, Lewis et al. at Facebook AI published "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". The idea was elegant: bolt a search engine onto a language model. For every question, search a knowledge base first, then feed the top results to the model along with the question.

By 2024 every serious LLM product used RAG: ChatGPT for Business, Microsoft Copilot, Perplexity, Notion AI, Glean, Klarna's customer service, Stripe's docs assistant. By 2026 it is the default architecture for LLM apps with private data.

The Mental Model

Five terms you must internalise:

Anatomy: Follow One Query

Let us trace a single query end-to-end. The user asks: "What is IITS's refund policy?"

Hands-On Level 1 — Toy Example

The minimum viable RAG. Twenty lines. In-memory. No vector database. Run it on your laptop right now.

import os, numpy as np
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

DOCS = [
    "Our refund policy allows returns within 30 days of purchase.",
    "Shipping is free for orders over 50 euros within the Netherlands.",
    "Customer support is available Monday to Friday, 9:00 to 17:00 CET.",
    "We accept iDEAL, credit card, and SEPA bank transfers.",
]

def embed(text):
    return client.embeddings.create(input=text, model="text-embedding-3-small").data[0].embedding

doc_vecs = np.array([embed(d) for d in DOCS])
question = "Can I return something I bought last week?"
q_vec = np.array(embed(question))

# Cosine similarity — find the most relevant doc
scores = doc_vecs @ q_vec / (np.linalg.norm(doc_vecs, axis=1) * np.linalg.norm(q_vec))
top_doc = DOCS[int(np.argmax(scores))]

prompt = f"Context: {top_doc}\n\nQuestion: {question}\n\nAnswer using only the context above."
answer = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}])
print(answer.choices[0].message.content)

Hands-On Level 2 — Realistic

Level 1 collapses at four problems: (a) the documents do not fit in memory, (b) there is no chunking, (c) there is no re-ranking, (d) the prompt does not enforce grounding. Level 2 fixes all four with ChromaDB and Cohere Rerank.

import os, chromadb, cohere
from openai import OpenAI
from langchain_text_splitters import RecursiveCharacterTextSplitter

openai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
co = cohere.Client(os.environ["COHERE_API_KEY"])
db = chromadb.PersistentClient(path="./rag_db").get_or_create_collection("docs")

# 1. Chunk a real document (recursive splitter respects paragraphs and sentences)
with open("policy.md") as f:
    text = f.read()
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_text(text)

# 2. Embed and store
vectors = [openai.embeddings.create(input=c, model="text-embedding-3-small").data[0].embedding for c in chunks]
db.add(ids=[str(i) for i in range(len(chunks))], embeddings=vectors, documents=chunks)

def ask(question):
    # 3. Retrieve top-20 candidates
    q_vec = openai.embeddings.create(input=question, model="text-embedding-3-small").data[0].embedding
    results = db.query(query_embeddings=[q_vec], n_results=20)
    candidates = results["documents"][0]

    # 4. Re-rank to top-3
    reranked = co.rerank(query=question, documents=candidates, top_n=3, model="rerank-english-v3.0")
    top_chunks = [candidates[r.index] for r in reranked.results]

    # 5. Augmented prompt with grounding instruction
    context = "\n\n---\n\n".join(top_chunks)
    prompt = (
        "Answer the question using ONLY the context below. "
        "If the answer is not in the context, say 'I don't have that information.' "
        f"Cite the relevant chunk number.\n\nContext:\n{context}\n\nQuestion: {question}"
    )
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return response.choices[0].message.content

print(ask("What is our refund policy for digital products?"))

Hands-On Level 3 — Production Pattern

In production you separate concerns, add observability, evaluate quality, and add hybrid (vector + keyword) search. Sketch of a production-shaped module:

# rag/pipeline.py
from dataclasses import dataclass
from typing import Sequence
import logging, time

@dataclass
class RetrievedChunk:
    text: str
    score: float
    source: str
    chunk_id: str
    metadata: dict

class RAGPipeline:
    def __init__(self, embedder, vector_store, keyword_store, reranker, llm, *, top_k=20, top_n=4):
        self.embedder = embedder
        self.vector_store = vector_store      # e.g. Azure AI Search, pgvector
        self.keyword_store = keyword_store    # e.g. BM25 over the same corpus
        self.reranker = reranker              # cross-encoder
        self.llm = llm
        self.top_k = top_k
        self.top_n = top_n
        self.log = logging.getLogger("rag")

    def ask(self, question: str, *, tenant_id: str, trace_id: str) -> dict:
        t0 = time.perf_counter()

        # 1. Hybrid retrieval — vector + keyword in parallel
        q_vec = self.embedder.embed(question)
        vec_hits = self.vector_store.search(q_vec, top_k=self.top_k, filter={"tenant": tenant_id})
        kw_hits = self.keyword_store.search(question, top_k=self.top_k, filter={"tenant": tenant_id})
        merged = self._reciprocal_rank_fusion(vec_hits, kw_hits)

        # 2. Re-rank
        reranked = self.reranker.rerank(question, merged, top_n=self.top_n)

        # 3. Fallback if nothing relevant
        if not reranked or reranked[0].score < 0.3:
            self.log.info("low_confidence_retrieval", extra={"trace_id": trace_id})
            return {"answer": "I don't have information on that.", "sources": [], "trace_id": trace_id}

        # 4. Augmented prompt
        context = self._format_context(reranked)
        prompt = self._build_prompt(question, context)
        answer = self.llm.generate(prompt, temperature=0, max_tokens=600)

        # 5. Observability
        self.log.info(
            "rag_query",
            extra={
                "trace_id": trace_id,
                "tenant_id": tenant_id,
                "latency_ms": int((time.perf_counter() - t0) * 1000),
                "n_retrieved": len(merged),
                "n_used": len(reranked),
                "top_score": reranked[0].score,
            },
        )

        return {
            "answer": answer,
            "sources": [{"id": c.chunk_id, "source": c.source, "score": c.score} for c in reranked],
            "trace_id": trace_id,
        }

Add an evaluation harness with RAGAS measuring faithfulness (is the answer grounded in the context?), answer relevancy, and context precision/recall. Run it on every change to your chunking, embeddings, or prompt.

Eight Pitfalls (Lessons from Production)

RAG vs Fine-Tuning vs Long Context vs Cache-Augmented

Cost & Performance Reality

Going Further

Curated Resources

• Lewis et al. (2020) — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (the original paper)
• Anthropic — Contextual Retrieval (a 2024 technique that cuts retrieval failures by 49%)
• Eugene Yan — Patterns for Building LLM-based Systems & Products (the most-cited practical guide)
• RAGAS — RAG evaluation library
• LlamaIndex docs (the most RAG-focused framework)
• Azure AI Search RAG overview (production patterns for Microsoft stack)

Self-Check: Have You Learned It?

chunks = db.query(query_embeddings=[q], n_results=50)
context = "\n".join(chunks)
prompt = f"Answer: {question}\nContext: {context}"

Frequently Asked Questions