RAG Explained: How to Give Your LLM a Memory It Can Actually Trust

You're building a chatbot for a company's internal documentation — hundreds of pages of engineering specs, HR policies, product guides. You plug in ChatGPT, write a clean system prompt,launch a demo. Someone asks "What's the reimbursement limit for travel expenses?" and the model confidently answers with a number. The number is wrong. It's not from your docs — it's from the model's general training data, some pattern absorbed from the internet years ago that roughly sounds like a reasonable reimbursement policy.

This is the fundamental problem with raw LLMs in production: they don't know your data. They know what was in their training set — a snapshot of text from months or years ago, with no awareness of your internal docs, your product specs, your latest pricing, your customer records, or anything else that lives inside your organization. And when they don't know something, they don't say "I don't know" — they say something plausible-sounding that might be completely fabricated.

The fix has a name: Retrieval-Augmented Generation, almost universally abbreviated as RAG. The idea is elegant: before asking the model to answer a question, find the relevant pieces of your actual data and hand them to the model as context. Now, instead of guessing, the model is summarizing and reasoning over the real information you provided.

The gap between "RAG sounds simple" and "RAG actually works reliably in production" is where most teams spend weeks of engineering time. Understanding the full pipeline — every
stage, every design decision, every place it quietly breaks — is what this article is about.

What We'll Cover

1. The RAG Mental Model — Why It Works at All
2. The Full RAG Development Pipeline (Both Phases)
3. Document Loaders — Getting Your Data In
4. Text Splitting and Chunking — The Stage Everyone Underestimates
5. Embeddings and Vector Stores — Where Your Data Lives at Query Time
6. Retrievers — Finding the Right Chunks When It Matters
7. Putting It All Together — A Working RAG Chain
8. Where RAG Breaks and How to Diagnose It

---

Chapter 1: The RAG Mental Model — Why It Works at All

Before code, you need the right mental model, because RAG is counterintuitive until it clicks.

Think about how you answer questions as a consultant. If someone asks you a question about a client's contract terms, you don't try to recall everything from memory — you pull up the contract, skim for the relevant section and answer based on what's in front of you. You're not generating from memory; you're reading and summarizing from a source.

RAG turns an LLM into that consultant. You build a system that:

1. Ingests your documents (PDFs, web pages, databases, whatever) and stores them in a searchable format.
2. When a user asks a question, retrieves the most relevant pieces of those documents.
3. Augments the model's prompt with those retrieved pieces.
4. Lets the model generate an answer based on the context you just handed it. The LLM's role shifts from "know everything" to "read what I gave you and reason over it." That's a job LLMs are genuinely very good at.

Without RAG:
User question  →  LLM (guesses from training data)  →  Response (possibly hallucinated)

With RAG:
User question  →  Retrieve relevant docs  →  LLM (reads your actual docs)  →  Grounded response

Enter fullscreen mode Exit fullscreen mode

This is why RAG is the dominant pattern for enterprise AI applications. It doesn't eliminate the LLM's tendency to hallucinate — but it gives the model true ground to stand on,and makes it much easier to audit: if the answer is wrong, you can trace it back to exactly which chunk was retrieved and why.

The full pipeline has two distinct phases that most beginners conflate into one. Let's separate them.

---

Chapter 2: The Full RAG Development Pipeline — Two Phases

Almost every RAG confusion I've seen in teams comes from mixing up what happens before a user ever sends a message versus what happens when they do. These are two fundamentally different processes with different triggers, different costs, and different failure modes.

Phase 1 — Indexing (Offline, Runs Once)

This is the preparation phase. You run it ahead of time — sometimes once, sometimes on a schedule, sometimes triggered by new document uploads. It does not happen in response to user queries.

[ Your Raw Documents ]
        │
        ▼
[ Document Loader ]      ← Pull documents in from wherever they live
        │
        ▼
[ Text Splitter ]        ← Break documents into manageable chunks
        │
        ▼
[ Embedding Model ]      ← Convert each chunk into a numeric vector
        │
        ▼
[ Vector Store ]         ← Store vectors + original text in a searchable database

Enter fullscreen mode Exit fullscreen mode

The result: a vector store that contains your entire knowledge base in a format where "find me content similar to this question" is a fast, cheap operation.

Phase 2 — Retrieval + Generation (Online, Runs Per Query)

This is the live path. It runs every time a user sends a message.

[ User Question ]
        │
        ▼
[ Embedding Model ]      ← Convert the question into a vector
        │
        ▼
[ Vector Store Retriever ]  ← Find the chunks most similar to the question
        │
        ▼
[ Prompt Template ]      ← Inject retrieved chunks + question into a prompt
        │
        ▼
[ LLM ]                  ← Generates answer grounded in the retrieved context
        │
        ▼
[ Response to User ]

Enter fullscreen mode Exit fullscreen mode

These two phases have completely different performance characteristics. Indexing is slow and expensive per document but runs offline. Retrieval + generation must be fast — it's the live user experience. Every design decision in RAG falls into one of these two phases. Keep this separation in mind as we go through each component.

---

Chapter 3: Document Loaders — Getting Your Data In

The first problem in any real RAG system is always the same: your data doesn't live in a clean text file. It lives in PDFs, Word documents, Notion pages, Confluence wikis,Google Drive folders, databases, Slack threads, web pages, and half a dozen other formats
— each requiring different handling to extract usable text.

Document loaders in LangChain are the abstraction that handles this — they give you a consistent load() interface regardless of the source, returning a list of Document objects (each containing a page_content string and a metadata dict).

from langchain_community.document_loaders import (
    PyPDFLoader,         # PDFs — extracts text page by page
    WebBaseLoader,       # Web pages — fetches and strips HTML
    TextLoader,          # Plain .txt files
    CSVLoader,           # Spreadsheets — each row becomes a Document
    UnstructuredWordDocumentLoader,  # .docx files
    NotionDirectoryLoader,           # Exported Notion workspace
)

# Every loader shares the same interface — .load() returns List[Document]

# Load a PDF — returns one Document per page, with page number in metadata
pdf_loader = PyPDFLoader("engineering_spec.pdf")
pdf_docs = pdf_loader.load()
# pdf_docs[0].page_content → "Chapter 1: System Architecture..."
# pdf_docs[0].metadata     → {"source": "engineering_spec.pdf", "page": 0}

# Load a web page — strips HTML, extracts main text content
web_loader = WebBaseLoader("https://docs.mycompany.com/api-reference")
web_docs = web_loader.load()

# Load a CSV — useful for structured data like FAQs or product catalogs
csv_loader = CSVLoader("product_faq.csv", source_column="url")
csv_docs = csv_loader.load()

Enter fullscreen mode Exit fullscreen mode

The metadata dict is not an afterthought — it's critical for production systems. Every document you ingest should carry metadata: its source URL, file name, creation date, author, document type, whatever is relevant. You'll use this metadata later for filtering ("only retrieve from documents tagged HR policy") and for citations in your responses ("according to engineering_spec.pdf, page 12").

# Always inspect what your loader actually produced before moving on.
# Loaders fail silently in surprising ways — PDFs with scanned images return empty
# page_content strings; HTML with JavaScript-rendered content may come back blank.
for doc in pdf_docs[:3]:
    print(f"Content length: {len(doc.page_content)} chars")
    print(f"Metadata: {doc.metadata}")
    print(f"Preview: {doc.page_content[:200]}")
    print("---")

Enter fullscreen mode Exit fullscreen mode

⚠️ Heads up: PDFs are the most common source and the most problematic. Scanned PDFs
(images of text rather than actual text) will return empty strings with most loaders —
you'll need OCR processing before the text is even extractable. Always validate that
your loader is actually getting text, not silence.

---

Chapter 4: Text Splitting and Chunking — The Stage Everyone Underestimates

Here's where most RAG beginners make their biggest architectural mistake, because this stage looks trivial and isn't.

You have your documents loaded. They might be entire 80-page PDFs or multi-thousand-word web pages. You can't stuff all of that into a prompt — LLMs have context limits, and even if you could, you don't want to. You want to retrieve the specific relevant section, not an entire document.

So you split documents into chunks — smaller pieces of text that can be independently retrieved and included in a prompt. The decisions you make here — how big each chunk is, how they overlap, how they respect the document's natural structure — have a bigger impact on RAG quality than almost any other single variable.

Chunk Size: The Core Trade-off

This is a genuine tension with no universal right answer:

• Too small (e.g., 100 tokens): Each chunk has very little context. You might retrieve a sentence that perfectly matches the query but doesn't contain enough surrounding information to answer it meaningfully. The model gets fragments.
• Too large (e.g., 2000 tokens): Each chunk contains more context, but semantic similarity search becomes less precise — a chunk about three different topics might score as a moderate match for many queries rather than a strong match for any. You also use more of your LLM's context window per retrieved chunk.
• Sweet spot in practice: 200–500 tokens per chunk for most knowledge-base use cases, with experimentation required for your specific content type and query patterns.

Chunk Overlap: Handling Boundary Problems

When you split at token/character boundaries, you'll slice through sentences, sometimes mid-thought. The sentence that contains the key information might end up half in chunk 12 and half in chunk 13, and neither chunk retrieves well. Overlap addresses this by
repeating the last N tokens of each chunk at the start of the next.

from langchain.text_splitter import RecursiveCharacterTextSplitter

# The workhorse of LangChain splitting — tries to split on natural boundaries
# first (paragraphs → sentences → words) before falling back to raw characters.
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,      # Target max characters per chunk
    chunk_overlap=50,    # Last 50 chars of chunk N repeated at start of chunk N+1
    separators=["\n\n", "\n", ". ", " ", ""]  # Try these in order
)

chunks = splitter.split_documents(pdf_docs)

print(f"Original: {len(pdf_docs)} pages")
print(f"After splitting: {len(chunks)} chunks")

# Good habit: sanity-check your chunks before continuing
for i, chunk in enumerate(chunks[:3]):
    print(f"\n--- Chunk {i} ({len(chunk.page_content)} chars) ---")
    print(chunk.page_content)

Enter fullscreen mode Exit fullscreen mode

Splitting Strategies: Not All Text Is Flat Prose

RecursiveCharacterTextSplitter is the general-purpose default, but different content
types benefit from different strategies:

from langchain.text_splitter import (
    RecursiveCharacterTextSplitter,   # General prose — paragraphs, articles, docs
    MarkdownHeaderTextSplitter,       # Markdown — splits on headings, preserves structure
    Language,                         # Code — splits on functions/classes, not mid-line
)

# For technical documentation in Markdown — splits on headers rather than character count
md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#", "h1"),
        ("##", "h2"),
        ("###", "h3"),
    ]
)
# Each resulting chunk carries header metadata — great for citations later

# For code repositories — respects function/class boundaries
code_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=1000,
    chunk_overlap=100,
)

Enter fullscreen mode Exit fullscreen mode

💡 Pro tip: Add the source metadata to every chunk explicitly — once you've split
200 documents into 5,000 chunks, you need to know which chunk came from where.
LangChain's splitters preserve metadata from the parent Document, but always verify.

---

Chapter 5: Embeddings and Vector Stores — Where Your Data Lives

You now have thousands of text chunks. The next problem: how do you find the chunks relevant to a specific user question, fast, at query time?

The answer is semantic search, and it works through embeddings.

What Embeddings Actually Are

Remember from the first article: each chunk of text can be converted into a vector — a list of hundreds or thousands of numbers — that represents its meaning in mathematical space. This conversion is done by an embedding model (a different, smaller model than your LLM — its only job is text → vector).

The key property: semantically similar text produces mathematically similar vectors. "How do I reset my password?" and "Steps to change my account credentials" are worded completely differently but mean similar things — their embedding vectors will be close together in vector space, close enough to find each other via similarity search.

This is why RAG can retrieve relevant content even when the user's question uses completely different words from your documentation. You're not searching for matching keywords; you're searching for matching meaning.

from langchain_openai import OpenAIEmbeddings
from langchain_community.embeddings import HuggingFaceEmbeddings

# OpenAI's embedding model — cloud API, very good quality
openai_embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Or a local HuggingFace model — free, runs on your hardware, great for cost-sensitive
# or privacy-sensitive use cases (your documents never leave your infrastructure) local_embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")

# What actually happens under the hood (you usually don't call this directly):
vector = openai_embeddings.embed_query("How do I reset my password?")
# vector is a list of ~1500 floats — the "meaning fingerprint" of that sentence print(f"Vector dimensions: {len(vector)}")  # e.g., 1536 for text-embedding-3-small

Enter fullscreen mode Exit fullscreen mode

Vector Stores

Embeddings are only useful if you can search through them efficiently. A vector store is a database built for this — you insert vectors + their associated text, and at query time you give it a query vector and it returns the N most similar stored vectors.

from langchain_community.vectorstores import (
    FAISS,       # Facebook AI Similarity Search — in-memory, great for dev/small scale
    Chroma,      # Lightweight persistent vector store — good for local development
    Pinecone,    # Managed cloud vector database — production scale
    Weaviate,    # Open-source cloud/self-hosted — good for complex filtering
    pgvector,    # Postgres extension — great if you're already on Postgres
)

# Building a FAISS vector store from chunks — this runs your embedding model
# on every chunk and stores the result. This is the slow, expensive indexing step. 
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

# This call: embeds every chunk + builds the search index
vectorstore = FAISS.from_documents(
    documents=chunks,      # Your split documents from Chapter 4
    embedding=embeddings   # The model that converts text → vector
)

# Persist to disk so you don't re-embed every time your app restarts
vectorstore.save_local("my_knowledge_base")

# Load it back later — no re-embedding needed
vectorstore = FAISS.load_local("my_knowledge_base", embeddings,
                                allow_dangerous_deserialization=True)

Enter fullscreen mode Exit fullscreen mode

⚠️ Heads up: Embedding your entire document set costs money and takes time —
OpenAI's embedding API charges per token, and embedding 10,000 chunks can take minutes.
Always persist your vector store to disk (or use a managed service) so the indexing
phase runs once, not on every app restart.

---

Chapter 6: Retrievers — Finding the Right Chunks When It Matters

A retriever is the interface that sits in front of your vector store and answers one question: given this query, what chunks should I surface? It's what Phase 2 of the pipeline calls when a user sends a message.

The simplest retriever is just a vector store wrapper:

# The most basic retriever — pure semantic similarity search
retriever = vectorstore.as_retriever(
    search_type="similarity",   # Find the N most semantically similar chunks
    search_kwargs={"k": 4}      # Return the top 4 chunks
)

# Test it — this is what runs at query time:
relevant_chunks = retriever.invoke("What is the reimbursement limit for travel?")
for chunk in relevant_chunks:
    print(chunk.page_content[:300])
    print(f"Source: {chunk.metadata.get('source', 'unknown')}")
    print("---")

Enter fullscreen mode Exit fullscreen mode

But "top 4 by similarity" is actually a pretty naive strategy, and there are several retrieval approaches that meaningfully improve quality in practice.

MMR — Maximum Marginal Relevance

Pure similarity search has a hidden flaw: if your knowledge base has five paragraphs that all say the same thing slightly differently, you'll retrieve all five of them instead of
getting diverse, complementary information. MMR trades off between relevance (is this chunk similar to the query?) and diversity (is this chunk different from what I've already retrieved?).

retriever = vectorstore.as_retriever(
    search_type="mmr",          # Maximum Marginal Relevance
    search_kwargs={
        "k": 4,                  # Return 4 chunks
        "fetch_k": 20,           # Consider top 20 by similarity first
        "lambda_mult": 0.7       # 0 = pure diversity, 1 = pure similarity
    }
)

Enter fullscreen mode Exit fullscreen mode

Similarity Score Threshold

Don't retrieve bad chunks just to hit your k target. If nothing in your knowledge base is actually relevant to the query, it's better to return nothing than to return vaguely-related content that confuses the model.

retriever = vectorstore.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={
        "score_threshold": 0.7,  # Only return chunks above this similarity score
        "k": 4
    }
)

Enter fullscreen mode Exit fullscreen mode

Metadata Filtering

Vector search finds semantically similar content, but sometimes you need to constrain which documents are even eligible — only search HR policies, or only search documents updated in the last 30 days, or only search content in English. This is where metadata filters come in.

retriever = vectorstore.as_retriever(
    search_kwargs={
        "k": 4,
        "filter": {"department": "HR", "document_type": "policy"}
        # Only consider chunks where metadata matches these conditions
    }
)

Enter fullscreen mode Exit fullscreen mode

Multi-Query Retrieval

One underrated technique: the user's question, as phrased, might not be the best query for semantic search. Multi-query retrieval uses the LLM itself to generate multiple alternative phrasings of the question, runs all of them as separate queries, and takes the union of results. This catches relevant chunks that a single query might miss due to phrasing mismatches.

from langchain.retrievers import MultiQueryRetriever
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(temperature=0)

multi_retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
    llm=llm
    # Under the hood: generates 3-5 alternative phrasings of the query,
    # retrieves for each, deduplicates results
)

Enter fullscreen mode Exit fullscreen mode

---

Chapter 7: Putting It All Together — A Working RAG Chain

Here's a complete, minimal, working RAG chain using LangChain's modern interface. This is the skeleton of nearly every document Q&A system you'll ever build.

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# ─── PHASE 1: INDEXING (run once, offline) ───────────────────────────────────

# 1. Load documents
loader = PyPDFLoader("company_handbook.pdf")
raw_docs = loader.load()

# 2. Split into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(raw_docs)

# 3. Embed and store
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)
vectorstore.save_local("handbook_index")

# ─── PHASE 2: RETRIEVAL + GENERATION (runs per query) ────────────────────────

# Load the pre-built index (skip re-embedding)
vectorstore = FAISS.load_local("handbook_index", embeddings,
                                allow_dangerous_deserialization=True)

retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

# The prompt that instructs the model to answer ONLY from retrieved context
rag_prompt = ChatPromptTemplate.from_messages([
    ("system",
     "You are a helpful assistant answering questions about company policy. "
     "Use ONLY the following retrieved context to answer the question. "
     "If the answer is not in the context, say: 'I don't have that information.' "
     "Do not use any knowledge outside of what's provided below.\n\n"
     "Context:\n{context}"),
    ("user", "{question}")
])

llm = ChatOpenAI(model="gpt-4o", temperature=0)  # temp=0 for consistent, factual answers

def format_docs(docs):
    # Combine retrieved chunks into a single context string
    return "\n\n".join(
        f"[Source: {doc.metadata.get('source', 'unknown')}, "
        f"Page: {doc.metadata.get('page', '?')}]\n{doc.page_content}"
        for doc in docs
    )

# Build the chain using LangChain Expression Language (LCEL)
rag_chain = (
    {
        "context": retriever | format_docs,  # retrieve → format as string
        "question": RunnablePassthrough()    # pass question through unchanged
    }
    | rag_prompt         # fill the template
    | llm                # call the model
    | StrOutputParser()  # parse response to plain string
)

# Use it
response = rag_chain.invoke("What is the reimbursement limit for travel expenses?") print(response)
# "According to the handbook, the travel reimbursement limit is $350 per night
#  for hotels and $75 per day for meals..."

Enter fullscreen mode Exit fullscreen mode

This chain is clean, auditable, and straightforward to swap out individual pieces — change the loader, the splitter settings, the retriever strategy, or the underlying LLM without touching the rest of the pipeline.

---

Chapter 8: Where RAG Breaks and How to Diagnose It

RAG systems fail in consistent, diagnosable ways. Knowing the failure modes ahead of time saves days of debugging.

"The model ignores the retrieved context and makes things up anyway." Your system prompt isn't firm enough. Add explicit grounding instructions: "Answer only from the context below. If the answer is not in the context, say so." Setting temperature=0
on the LLM also helps with consistency.

"Retrieval returns irrelevant chunks."
This is the most common failure, and it almost always traces to chunking. Chunks are too large (low retrieval precision), too small (not enough context per chunk), or split at bad boundaries (sentences split mid-idea). Try smaller chunks, increase overlap, or switch to a structure-aware splitter (markdown header, code-aware).

"The right information is in the document but retrieval never finds it." Your query phrasing and your document phrasing are too semantically distant. Try the multi-query retriever to cast a wider net. Also check: was this section actually indexed? Loaders silently fail on scanned PDFs, complex tables, and certain layouts.

"The model's answer is correct but I can't trace where it came from." Always include source metadata in your formatted context and instruct the model to cite it. Without attribution, RAG is a black box — with attribution, it's auditable.

"It works in testing but gives bad answers on long documents at scale." Your k value (number of retrieved chunks) is hitting the LLM's context window limit. Either reduce k, reduce chunk size, or use a model with a larger context window. Also check for embedding model mismatch — if you indexed with one embedding model and query with another, the vector space is incompatible.

---

Where This Takes You Next

You now have a mental model of the complete RAG pipeline from ingestion to response — not just what the pieces are called, but why each one exists and what breaks when it's wrong.

The natural next step is building something real with this. Start with a single clean PDF and a handful of test questions whose correct answers you already know. Run the full pipeline, check whether your retriever surfaces the right chunks for each question, then tune chunk size and overlap until retrieval quality is solid. Get that working before touching retriever strategies or advanced techniques.

Once retrieval is reliable, the rest — getting the model to answer well, adding conversation memory, citing sources — falls into place quickly. The retrieval quality is the foundation. Everything else is downstream from it.

RAG isn't magic. It's plumbing — well-designed plumbing that lets a language model answer questions about your actual data instead of guessing. Build the plumbing carefully, and the magic takes care of itself.