From documents to embeddings, retrieval, reranking, prompt construction, and grounded answers — a practical walkthrough of how modern RAG pipelines actually work.
Introduction
In the previous article, we looked at Why LLMs Hallucinate is and why RAG has become one of the most important architectures for building LLM applications.
But knowing what RAG is isn’t enough.
The real question is:
What actually happens between a user’s question and the final answer in a RAG system?
Suppose an employee asks an internal AI assistant:
“How many days of parental leave are employees entitled to?”
The LLM itself may not have access to the company’s latest HR policy.
A RAG system solves this by finding the relevant information first and then giving that information to the LLM.
At a high level:
User Question ↓Retrieve relevant information ↓Add information to the prompt ↓LLM generates answer
But production RAG systems are considerably more sophisticated.
A typical pipeline looks like this:
┌─────────────────────┐
│ Company Documents │
│ PDFs / Web / DB / │
│ Docs / APIs │
└──────────┬──────────┘
│
▼
Document Parsing
│
▼
Chunking
│
▼
Embeddings
│
▼
Vector Database
│
│
───────────┼────────────
│
User Question
│
▼
Query Embedding
│
▼
Retrieval
│
▼
Reranking
│
▼
Context Construction
│
▼
LLM
│
▼
Final Response
This article breaks down every stage.
Table of Contents
- What You’ll Learn
- Prerequisites
- The Two Phases of RAG
- Phase 1 — Building the Knowledge Base
- Document Ingestion
- Document Chunking
- Generating Embeddings
- Storing Vectors
- Phase 2 — Answering a User Question
- Query Processing
- Retrieval
- Reranking
- Prompt Construction
- Generation
- Complete RAG Example in Python
- RAG Architecture
- Why Retrieval Quality Matters
- Common RAG Mistakes
- Production Considerations
- Interview Questions
- Key Takeaways
- Related Tutorials
- Next Tutorial
What You’ll Learn
By the end of this tutorial, you’ll understand:
- How documents enter a RAG system
- Why documents need to be chunked
- What embeddings do
- How vector databases perform semantic retrieval
- How a user query is processed
- Why reranking is useful
- How retrieved context reaches the LLM
- How the final answer is generated
- How a basic RAG pipeline can be implemented in Python
- What changes when moving from a prototype to production
Prerequisites
You should have a basic understanding of:
- Python
- Machine learning fundamentals
- LLMs
- Embeddings
- APIs
- Basic NLP concepts
You don’t need to understand the mathematics of Transformers to follow this tutorial.
The Two Phases of RAG
One of the easiest ways to understand RAG is to split it into two separate workflows.
Phase 1: Indexing
Prepare your knowledge base.
Documents ↓Parsing ↓Chunking ↓Embedding ↓Vector Database
This usually happens offline or asynchronously.
Phase 2: Retrieval + Generation
Answer user questions.
Question ↓Query Processing ↓Retrieval ↓Reranking ↓Prompt ↓LLM ↓Answer
This happens at query time.
Keeping these two phases separate makes RAG much easier to reason about.
Phase 1: Building the Knowledge Base
Let’s assume we’re building an AI assistant for a company’s HR department.
Our knowledge base contains:
employee_handbook.pdfleave_policy.pdfinsurance_policy.pdfremote_work_policy.pdf
The first task is getting useful text out of these documents.
Step 1: Document Ingestion
Documents can come from many sources:
PDFWordHTMLMarkdownDatabaseConfluenceSharePointS3APIs
For example:
documents = [ "Employees receive 20 days of annual leave.", "Employees can work remotely up to three days per week.", "Parental leave is available to eligible employees."]
A real application would use document loaders rather than hardcoding strings.
The important idea is:
RAG starts with your knowledge, not with the LLM.
Step 2: Chunking
A 200-page PDF shouldn’t normally be sent directly to an LLM every time someone asks a question.
Instead, we divide it into smaller pieces.
For example:
100-page document ↓┌───────────────┐│ Chunk 1 │├───────────────┤│ Chunk 2 │├───────────────┤│ Chunk 3 │├───────────────┤│ ... │├───────────────┤│ Chunk 500 │└───────────────┘
Why?
Because retrieval works at the chunk level.
Suppose a 100-page employee handbook contains one paragraph about parental leave.
We don’t want to retrieve all 100 pages.
We want something like:
Chunk 183──────────────Parental leave is available to eligiblefull-time employees...
That is much more useful to the LLM.
Chunk Size Is a Design Decision
Suppose we use:
chunk_size = 500chunk_overlap = 50
The overlap helps preserve context between neighboring chunks.
Conceptually:
Chunk 1────────────────────────A B C D E F G H │ └──── overlapChunk 2──────────────────────── G H I J K L M
But there is no universally optimal chunk size.
Too large:
Large chunk ↓Lots of irrelevant information ↓Poor context quality
Too small:
Tiny chunk ↓Missing context ↓Incomplete answer
Chunking deserves its own optimization process.
We’ll explore it in a dedicated tutorial later.
Step 3: Generate Embeddings
Now we need to make our chunks searchable by meaning.
Consider:
"Employees receive 20 days of annual leave."
An embedding model converts this sentence into a vector:
[0.21, -0.44, 0.73, 0.11, ...]
The exact numbers aren’t important.
What’s important is that semantically similar text tends to occupy nearby regions of the embedding space.
For example:
"How many vacation days do employees get?" ● Query"Employees receive 20 days of annual leave." ● Document
These should have similar embeddings.
Python: Generate Embeddings
We can use Sentence Transformers:
from sentence_transformers import SentenceTransformermodel = SentenceTransformer("all-MiniLM-L6-v2")chunks = [ "Employees receive 20 days of annual leave.", "Employees can work remotely up to three days per week.", "Parental leave is available to eligible employees."]embeddings = model.encode(chunks)print(embeddings.shape)
Each chunk now has a numerical representation.
Step 4: Store Embeddings
We need somewhere to store and search these vectors.
Popular vector databases and vector search systems include:
- FAISS
- Pinecone
- Qdrant
- Milvus
- Weaviate
- pgvector
For a simple local example, let’s use FAISS.
import faissimport numpy as npvectors = np.array(embeddings).astype("float32")dimension = vectors.shape[1]index = faiss.IndexFlatL2(dimension)index.add(vectors)
Now our vector index contains our document chunks.
But there’s an important problem.
The vector database doesn’t automatically know which vector belongs to which document.
We therefore maintain metadata.
metadata = [ { "text": "Employees receive 20 days of annual leave.", "source": "leave_policy.pdf", "page": 4 }, { "text": "Employees can work remotely up to three days per week.", "source": "remote_work_policy.pdf", "page": 2 }, { "text": "Parental leave is available to eligible employees.", "source": "leave_policy.pdf", "page": 7 }]
This metadata becomes extremely important in production.
Phase 2: Answering a User Question
Now suppose someone asks:
“How many vacation days do employees get?”
The question itself doesn’t directly go to the LLM.
It first goes through retrieval.
Step 5: Embed the User Query
We use the same embedding model.
query = "How many vacation days do employees get?"query_embedding = model.encode([query])
Now both the question and documents live in the same vector space.
Step 6: Retrieve Relevant Chunks
We search the vector index.
query_vector = np.array(query_embedding).astype("float32")distances, indices = index.search( query_vector, k=2)
k=2 means:
Return the two most similar chunks.
We can then retrieve the corresponding documents.
for i in indices[0]: print(metadata[i]["text"])
Potential output:
Employees receive 20 days of annual leave.Parental leave is available to eligible employees.
Now we have relevant context.
But Retrieval Isn’t the Same as Answering
This distinction is extremely important.
The vector database doesn’t answer the question.
It only says:
“These documents appear relevant.”
The LLM is still responsible for generating the answer.
So the pipeline is:
Question ↓Retriever ↓Relevant Context ↓LLM ↓Answer
Step 7: Reranking
A production RAG system often adds another step.
Instead of sending every retrieved chunk to the LLM, we can rerank them.
For example:
User Query ↓Vector Search ↓Top 20 chunks ↓Reranker ↓Top 5 chunks ↓LLM
Why?
Vector similarity is useful for finding candidates, but it isn’t always the best final relevance signal.
A cross-encoder reranker can examine the query and candidate text together.
Conceptually:
Query + Chunk ↓Relevance Model ↓Score
This can improve context quality before generation.
We’ll cover cross-encoder reranking in a later tutorial.
Step 8: Construct the Prompt
Now we have the relevant documents.
We insert them into the prompt.
context = """Employees receive 20 days of annual leave."""prompt = f"""You are an HR assistant.Answer the question using only the provided context.Context:{context}Question:How many vacation days do employees get?If the answer cannot be found in the context,say that the information is unavailable."""
This is the critical bridge between retrieval and generation.
Step 9: Generate the Answer
The prompt is sent to the LLM.
Conceptually:
┌─────────────────────────────────────┐│ System Instructions ││ ││ Context ││ Employees receive 20 days... ││ ││ User Question ││ How many vacation days...? │└──────────────────┬──────────────────┘ │ ▼ LLM │ ▼ "Employees receive 20 days of annual leave."
The model isn’t expected to retrieve the information itself.
The retrieval system has already done that work.
Complete RAG Architecture
We can now put everything together.
OFFLINE / INDEXING
┌───────────────────────────────────────────────┐
│ │
│ Documents │
│ │ │
│ ▼ │
│ Document Parser │
│ │ │
│ ▼ │
│ Chunking │
│ │ │
│ ▼ │
│ Embedding Model │
│ │ │
│ ▼ │
│ Vector Database + Metadata │
│ │
└───────────────────────┬───────────────────────┘
│
│
│
ONLINE
│
▼
User Question
│
▼
Query Embedding
│
▼
Vector Retrieval
│
▼
Reranking
│
▼
Context Selection
│
▼
Prompt Construction
│
▼
LLM
│
▼
Generated Answer
│
▼
Citation / Validation
This is the basic architecture behind many RAG applications.
A Minimal End-to-End Python Example
Let’s put the core pieces together.
import faissimport numpy as npfrom sentence_transformers import SentenceTransformer# -----------------------------# 1. Documents# -----------------------------documents = [ "Employees receive 20 days of annual leave.", "Employees can work remotely up to three days per week.", "Parental leave is available to eligible employees."]# -----------------------------# 2. Embedding Model# -----------------------------model = SentenceTransformer("all-MiniLM-L6-v2")# -----------------------------# 3. Create Document Embeddings# -----------------------------embeddings = model.encode(documents)vectors = np.array(embeddings).astype("float32")# -----------------------------# 4. Create Vector Index# -----------------------------dimension = vectors.shape[1]index = faiss.IndexFlatL2(dimension)index.add(vectors)# -----------------------------# 5. User Query# -----------------------------question = "How many vacation days do employees get?"query_embedding = model.encode([question])query_vector = np.array(query_embedding).astype("float32")# -----------------------------# 6. Retrieve Documents# -----------------------------distances, indices = index.search( query_vector, k=2)retrieved_documents = [ documents[i] for i in indices[0]]# -----------------------------# 7. Build Context# -----------------------------context = "\n".join(retrieved_documents)prompt = f"""Answer the question using only the following context.Context:{context}Question:{question}If the answer is not available in the context,say "I don't know.""""print(prompt)
At this point, you can pass prompt to your preferred LLM API.
The important thing is that the LLM receives retrieved evidence, rather than relying entirely on its pretrained knowledge.
Why RAG Works
RAG works because it separates two responsibilities.
Retrieval
Find the relevant information.
Generation
Turn that information into a useful response.
This is fundamentally different from asking an LLM to answer everything from its parameters.
Traditional LLMQuestion ↓Model Parameters ↓Answer
Versus:
RAGQuestion ↓External Knowledge ↓Relevant Context ↓LLM ↓Answer
The second architecture is much better suited to applications where knowledge changes frequently.
But RAG Doesn’t Automatically Solve Hallucinations
This is a critical point.
A common misconception is:
“If I use RAG, my LLM can’t hallucinate.”
That’s false.
Consider:
User Question ↓Retriever ↓Wrong Document ↓LLM ↓Confident Wrong Answer
The LLM can only work with the context it receives.
If retrieval is poor, generation can also be poor.
That’s why production RAG needs separate evaluation of:
Retrieval quality
and
Generation quality
Common RAG Failure Modes
1. Poor Chunking
Chunks are too large.
Result:
Relevant information+Lots of irrelevant information
The model has to process unnecessary context.
2. Chunks Are Too Small
Important context gets split across chunks.
Example:
Chunk 1:Employees are eligible for...Chunk 2:...20 days of leave after completing one year.
Retrieving only one chunk may produce an incomplete answer.
3. Wrong Embedding Model
Different embedding models are trained with different objectives and datasets.
A model that performs well for general semantic similarity may not be optimal for:
- legal documents
- medical documents
- code
- multilingual retrieval
Embedding model selection should therefore be evaluated against your actual retrieval workload.
4. Retrieving Too Many Chunks
You might retrieve:
Top 50
and send all of them to the LLM.
This can increase:
- token usage
- latency
- cost
- irrelevant context
More context isn’t necessarily better context.
5. Ignoring Metadata
Suppose you have:
Policy 2024Policy 2025Policy 2026
A semantic search might retrieve all three.
Metadata filtering can ensure that only the current policy is considered.
Production Considerations
A prototype RAG pipeline is easy.
Production RAG is a different problem.
You need to think about the entire system.
Latency
A request may involve:
Query embedding ↓Vector search ↓Reranking ↓LLM generation
Each stage adds latency.
You may need:
- caching
- parallel retrieval
- smaller embedding models
- optimized vector indexes
- streaming generation
Cost
LLM costs can increase rapidly if you’re sending large amounts of context.
Instead of:
50 chunks × 500 tokens
you might optimize toward:
5 relevant chunks × 300 tokens
while maintaining answer quality.
Security
Enterprise RAG systems can contain sensitive information.
You need:
- access control
- document-level permissions
- tenant isolation
- PII handling
- audit logging
A user shouldn’t retrieve a document simply because its embedding is semantically similar.
Authorization must happen independently of semantic relevance.
Freshness
Your knowledge base may change.
For example:
Policy v1 ↓Policy v2 ↓Policy v3
A production pipeline should handle:
- document versioning
- incremental indexing
- deletion
- re-indexing
- effective dates
Evaluation
Don’t evaluate only the final answer.
Measure different parts of the pipeline.
Retrieval
- Recall
- Precision
- MRR
- NDCG
Generation
- Faithfulness
- Answer relevance
- Citation correctness
System
- Latency
- Cost
- Throughput
- Failure rate
This separation makes debugging much easier.
RAG vs Fine-Tuning
A common question is:
“Why not just fine-tune the LLM on my documents?”
Because RAG and fine-tuning solve different problems.
| RAG | Fine-Tuning |
|---|---|
| Adds external knowledge | Changes model behavior |
| Easy to update | Requires training |
| Good for private documents | Good for specialized behavior |
| Can provide citations | Doesn’t inherently provide citations |
| Retrieval adds latency | Inference can be simpler |
| Knowledge stays outside model | Knowledge becomes encoded in weights |
For frequently changing enterprise information, RAG is often the better starting point.
Common Mistakes
Mistake 1: Starting with the LLM
Engineers sometimes immediately ask:
Which LLM should I use?
A better first question is:
What information does the application need to answer correctly?
Mistake 2: Treating RAG as a Vector Database
A vector database is only one component.
RAG includes:
Ingestion+Chunking+Embedding+Retrieval+Reranking+Prompting+Generation+Evaluation
Mistake 3: Optimizing Generation Before Retrieval
If the correct document isn’t retrieved, changing GPT models won’t necessarily solve the problem.
Fix retrieval first.
Mistake 4: Sending Everything to the LLM
More context isn’t always better.
You want:
The smallest amount of highly relevant context needed to answer the question.
Interview Questions
1. Explain RAG in one minute.
Answer:
Retrieval-Augmented Generation is an architecture where relevant information is retrieved from an external knowledge base and provided to an LLM as context before generating a response. It allows applications to use private, current, or domain-specific information without retraining the model.
2. What are the two main phases of RAG?
Answer:
The indexing phase prepares documents by parsing, chunking, embedding, and storing them. The query phase embeds the user’s question, retrieves relevant chunks, optionally reranks them, constructs a prompt, and sends the context to the LLM for generation.
3. Why do we need embeddings?
Answer:
Embeddings represent text as numerical vectors that capture semantic relationships. This allows the retrieval system to find documents that are conceptually similar to a query even when they don’t share the exact same keywords.
4. Why is reranking useful?
Answer:
Initial retrieval is optimized for efficiently finding candidate documents. A reranker can then perform a more precise relevance assessment on those candidates and select the most useful context for the LLM.
5. Why can RAG still hallucinate?
Answer:
RAG doesn’t guarantee correctness. If retrieval returns irrelevant or outdated information, or if the LLM fails to follow the provided context, the generated response can still be incorrect.
6. What would you optimize first if a RAG application produces poor answers?
Answer:
I’d first separate retrieval quality from generation quality. I’d inspect whether the correct chunks are being retrieved, then evaluate chunking, embedding quality, metadata filtering, and reranking before changing the LLM itself.
Key Takeaways
RAG isn’t simply:
Vector Database + LLM
It’s a complete information retrieval and generation pipeline.
The core workflow is:
Documents ↓Parsing ↓Chunking ↓Embeddings ↓Vector Database ↓User Query ↓Query Embedding ↓Retrieval ↓Reranking ↓Context ↓LLM ↓Answer
The most important lesson is:
A RAG application’s quality is constrained by the quality of the context it gives the LLM.
A powerful model cannot reliably answer a question if the correct information never reaches it.
That’s why strong AI engineers don’t treat RAG as an LLM problem alone.
They treat it as an end-to-end retrieval, ranking, generation, and evaluation problem.
Related Tutorials
This article belongs to the GeekyCodes RAG Fundamentals cluster.
RAG Fundamentals
- What Is RAG?
- Why LLMs Hallucinate?
- How RAG Works ← You are here
- Dense vs. Sparse Retrieval
- BM25 Explained
- Hybrid Search
- RAG Chunking Strategies
- Embeddings Explained
- Cross-Encoder Reranking
- Parent-Child Retrieval
- Multi-Query Retrieval
- Self-RAG
- Corrective RAG
- Graph RAG
- Agentic RAG
- RAG Evaluation
- Production RAG
Next Tutorial
The next logical step is to understand the retrieval layer in more detail.
Dense vs. Sparse Retrieval: Which Search Strategy Should You Use for RAG?
We’ll compare:
Keyword Search ↓BM25vs.Semantic Search ↓Embeddingsvs.Hybrid Search ↓BM25 + Vector Search
We’ll also build a practical Python example and understand why keyword search can still outperform semantic search for certain queries.
Series Navigation
← Previous
You are here:
RAG Fundamentals
Next →
[]
About the Author
Ved Prakash is a Senior Data Scientist and AI Engineer working across Machine Learning, Generative AI, LLMs, RAG, and production AI systems. He writes practical tutorials on GeekyCodes covering AI Engineering, Data Engineering, Machine Learning, and Generative AI.