Build a RAG Question-Answering System with Documents
Ask a plain chat model a question about your own manuals, contracts, or meeting notes and it will either invent an answer or tell you it cannot help. The fix is not a bigger model. The fix is retrieval-augmented generation, usually shortened to RAG, which pairs a search step with a generation step so the model answers using only the documents you actually gave it. In this tutorial you will build a working RAG question-answering system from scratch in Python, load your own PDFs and text files, and see the whole pipeline in about fifty lines of readable code. No framework required, though you can translate the same steps into LangChain or LlamaIndex afterward.
RAG sounds exotic, but it is really a three-part recipe: chunk your documents, embed them into a searchable space, and retrieve the most relevant pieces at question time to stuff into the prompt. Once you see those three steps spelled out, it stops being mysterious.
Understand the Three-Phase Pipeline
Every RAG system, no matter how elaborate, follows the same skeleton. Get this skeleton in your head first and the rest is implementation detail.
- Indexing: split raw documents into chunks, turn each chunk into a numeric vector with an embedding model, and store them in a vector database so they can be searched fast.
- Retrieval: when a question arrives, embed the question with the same model and find the most similar stored chunks by vector distance.
- Generation: paste the top chunks into the model's prompt as context, then ask it to answer using only that context.
The detail that trips up most beginners is this: the same embedding model must embed your documents and your question, or the vectors will live in different spaces and retrieval will be meaningless.
Get Your Environment Ready
You will need Python 3.9 or newer, an embedding provider plus a chat model provider, and the small set of libraries below. A free tier or a local model works fine for learning.
pip install openai chromadb pypdf
# or, for a fully local setup:
pip install sentence-transformers chromadb pypdf
I use Chroma here because it is a joy to install and runs fully in-process, which means no separate server process to babysit while you learn.
Step 1: Load and Chunk Your Documents
Start by reading your files and breaking them into digestible pieces. Chunking matters more than most people expect: too small and each chunk lacks context, too large and retrieval gets fuzzy and your prompt fills up fast.
from pypdf import PdfReader
def load_text(path):
reader = PdfReader(path)
return " ".join(p.extract_text() or "" for p in reader.pages)
def chunk(text, size=800, overlap=100):
words = text.split()
chunks = []
i = 0
while i < len(words):
piece = words[i:i+size]
chunks.append(" ".join(piece))
i += size - overlap
return chunks
The overlap is the trick to good chunking. By carrying a hundred words over into the previous chunk, you keep sentences and ideas that straddle a boundary intact instead of cutting them in half.
Step 2: Embed and Index
Now convert each chunk into a vector and push it into Chroma. You create a collection, then add the chunks with their IDs and any metadata you want to keep, like which file they came from.
import chromadb
from chromadb.utils import embedding_functions
ef = embedding_functions.OpenAIEmbeddingFunction(model="text-embedding-3-small")
client = chromadb.PersistentClient(path="./vec_db")
col = client.get_or_create_collection("docs", embedding_function=ef)
chunks = chunk(load_text("manual.pdf"))
col.add(
ids=[f"c{i}" for i in range(len(chunks))],
documents=chunks
)
Notice I called get_or_create_collection. That tiny choice makes your indexing idempotent, so re-running the script does not silently create duplicates, which is a classic footgun.
Step 3: Retrieve the Right Chunks
At question time you embed the question with the same function and ask the collection for the nearest neighbors. Chroma returns the top matches ordered by similarity.
def retrieve(question, k=4):
res = col.query(query_texts=[question], n_results=k)
return res["documents"][0]
A good retrieval returns four or five chunks, not fifty. You want just enough context to answer, with as little noise as possible. If your answers feel off, start by inspecting these retrieved chunks directly before blaming the model.
Step 4: Generate a Grounded Answer
Finally, paste the retrieved chunks into a system prompt that instructs the model to answer only from the context and admit when it does not know. This grounding prompt is the entire reason RAG reduces hallucination.
from openai import OpenAI
client = OpenAI()
def answer(question):
ctx = "\n---\n".join(retrieve(question))
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role":"system","content":
"Answer using ONLY the context. Say 'I do not know' if it is not there."},
{"role":"user","content":f"Context:\n{ctx}\n\nQuestion: {question}"},
]
)
return resp.choices[0].message.content
Run answer("What is the return policy?") on a policy manual and you should get an answer quoted from your document, not invented from nowhere. That shift, from plausible to grounded, is the whole point of RAG.
Leveling Up Your System
Once the basic loop works, there are three high-impact upgrades you can make in order of difficulty.
- Add metadata filters so retrieval only searches documents from a specific year or department, which improves both speed and accuracy dramatically.
- Use hybrid search, combining keyword matching with vector search, to catch exact terms that embeddings can occasionally blur.
- Add a reranker, a small model that re-scores the top candidates from coarse retrieval and picks the best, which is the single biggest accuracy win.
You do not need a bigger model to make RAG better. In most projects, the largest measured gains come from better chunking, metadata filters, and a reranker, not from swapping model sizes.
Troubleshooting Common Pitfalls
Choosing a Chunking Strategy That Fits Your Documents
Chunking is where many RAG pipelines quietly succeed or fail, and the right size depends far more on your documents than on any fixed default. Short chunks, around a few hundred characters, tend to retrieve more precisely but can lose the surrounding context needed for a complete answer. Long chunks keep more context but blur the boundary between topics and make similarity search return partially relevant lumps. The usual compromise is to size chunks around one coherent section or paragraph, then overlap the boundaries slightly so that a concept split across two chunks is not lost.
Whatever size you choose, keep the retrieval question in mind: the chunk should be exactly the unit that, when retrieved alone, lets the generation step answer confidently. For that reason, storing the source document and section along with each chunk is worthwhile, because returning a clean citation alongside the answer builds trust and lets users check the ground truth. A small experiment on a handful of your own questions will almost always beat a borrowed default, so budget a few minutes to tune chunk size before you lock it in.
Good retrieval is about returning the smallest possible unit that is still complete enough to support a grounded answer.
Finally, remember that chunking and retrieval quality are measured together. When a user asks about something spread across your corpus, the retriever's job is to surface the right pieces the generator can weave into one answer. Keep a handful of realistic questions as a retrieval benchmark and run it after every change to your chunking or embedding settings.
When your answers miss, check the boring things before the clever ones. A mismatch between the embedding model used for indexing and retrieval breaks everything silently, as does forgetting to re-index after your documents change. And if retrieval returns the wrong documents, look at your chunks: a good chunk should be a self-contained idea that still makes sense if you read it alone, which is exactly what grounding needs.
