Learn how RAG works, why LLMs hallucinate, and build your first Retrieval-Augmented Generation pipeline in Python.

Find all tutorials here

Introduction

Large Language Models (LLMs) have transformed how we build AI applications.

Today, we can ask models to:

• Summarize documents
• Write code
• Generate reports
• Answer questions
• Translate languages
• Create marketing content

Tools like ChatGPT, Claude, Gemini, and Llama make these tasks feel almost magical.

But there’s one major problem.

LLMs don’t actually know your data.

Ask ChatGPT:

“What is the leave policy at my company?”

or

“What does Clause 7.3 in our insurance policy say?”

The model has no idea.

Even worse, it may confidently invent an answer.

This phenomenon is called hallucination, and it’s one of the biggest challenges when deploying LLMs in production.

That’s where Retrieval-Augmented Generation (RAG) comes in.

Rather than relying solely on what the model learned during training, RAG retrieves relevant information from your own knowledge base before generating an answer.

Instead of guessing, the model answers using real documents.

In this tutorial, you’ll learn:

• What RAG is
• Why modern AI applications use it
• How the RAG pipeline works
• How to build a simple RAG system in Python
• Common mistakes beginners make
• How production RAG systems differ from toy examples

The Problem with Traditional LLMs

Imagine you’re building an AI assistant for a hospital.

Patients ask questions like:

What are the side effects of Drug X?

What is our insurance coverage?

What documents are needed before surgery?

A standalone LLM only knows information that existed during training.

It doesn’t know:

• your latest documents
• your internal knowledge base
• customer contracts
• PDFs
• policies
• manuals
• proprietary information

Even if the answer exists inside your company documents, the model cannot access it.

The result?

User Question
        │
        ▼
Large Language Model
        │
        ▼
❌ Makes an educated guess

Sometimes the guess is correct.

Sometimes it isn’t.

For healthcare, finance, insurance, and legal applications, guessing is unacceptable.

What Is RAG?

Retrieval-Augmented Generation combines two powerful systems:

1. Information Retrieval
2. Large Language Models

Instead of asking the LLM to answer from memory, we first retrieve relevant documents.

Only then does the model generate a response.

User Question
      │
      ▼
Retriever
      │
Relevant Documents
      │
      ▼
LLM
      │
      ▼
Grounded Answer

The retrieved documents become the model’s temporary memory.

This makes answers:

• more accurate
• more up-to-date
• explainable
• easier to trust

Why Is It Called Retrieval-Augmented Generation?

Let’s break the name down.

Retrieval

Search your knowledge base for relevant information.

Augmented

Add that information into the prompt.

Generation

Generate an answer using both the question and retrieved documents.

Hence:

Retrieval + Augmented + Generation

How Does a RAG Pipeline Work?

A production RAG system typically looks like this:

                    Documents
                        │
                        ▼
                  Data Ingestion
                        │
                        ▼
                   Text Extraction
                        │
                        ▼
                    Chunking
                        │
                        ▼
                  Embedding Model
                        │
                        ▼
                 Vector Database
──────────────────────────────────────────

                  User Question
                        │
                        ▼
                Query Embedding
                        │
                        ▼
                 Similarity Search
                        │
                        ▼
              Top-K Relevant Chunks
                        │
                        ▼
                 Prompt Builder
                        │
                        ▼
                  Large Language Model
                        │
                        ▼
                     Final Answer

This pipeline has two phases:

• Indexing (offline)
• Retrieval (online)

Let’s explore each.

Step 1: Load Documents

Suppose your company has:

• PDFs
• Word files
• Confluence pages
• FAQs
• Databases

These become your knowledge source.

Example:

documents = [
    "Employees receive 20 days of paid leave every year.",
    "Health insurance covers hospitalization expenses.",
    "Annual bonuses are paid in December."
]

Step 2: Split Documents into Chunks

LLMs cannot process huge documents efficiently.

Instead, documents are divided into smaller chunks.

Example:

100-page PDF
↓
Chunk 1
↓
Chunk 2
↓
Chunk 3
↓
...
↓
Chunk N

Chunking is one of the most important design decisions in RAG.

We’ll cover advanced chunking strategies in a dedicated article later in this series.

Step 3: Convert Chunks into Embeddings

Computers don’t understand text.

They understand numbers.

Each chunk is converted into a dense vector.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(documents)

Each document now becomes something like:

[0.24, -0.18, 0.91, ...]

Documents with similar meanings have similar vectors.

Step 4: Store Embeddings in a Vector Database

Now we save the embeddings.

Popular choices include:

• FAISS
• Pinecone
• Qdrant
• Milvus
• Weaviate
• pgvector

For this tutorial, we’ll use FAISS because it runs locally.

import faiss
import numpy as np
dimension = embeddings.shape[1]
index = faiss.IndexFlatL2(dimension)
index.add(np.array(embeddings).astype("float32"))

Our documents are now searchable.

Step 5: Convert the User Query into an Embedding

Suppose the user asks:

How many leave days do employees receive?

Convert the question into a vector.

query = "How many leave days do employees receive?"
query_vector = model.encode([query])

Step 6: Find Similar Documents

Search the vector database.

D, I = index.search(
    np.array(query_vector).astype("float32"),
    k=2
)

The index returns the most similar chunks.

Example:

Employees receive 20 days of paid leave every year.

Step 7: Build the Prompt

Instead of asking:

How many leave days do employees receive?

We ask:

Context:
Employees receive 20 days of paid leave every year.
Question:
How many leave days do employees receive?
Answer using only the provided context.

Now the LLM has evidence.

Step 8: Generate the Final Answer

The LLM responds:

Employees receive 20 days of paid leave every year, according to the provided company policy.

Notice something important.

The model isn’t inventing information.

It’s using retrieved knowledge.

Complete Example

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

documents = [
    "Employees receive 20 days of paid leave every year.",
    "Annual bonuses are paid in December.",
    "Health insurance covers hospitalization expenses."
]

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(documents)

index = faiss.IndexFlatL2(embeddings.shape[1])

index.add(np.array(embeddings).astype("float32"))

query = "How many leave days do employees receive?"

query_embedding = model.encode([query])

D, I = index.search(
    np.array(query_embedding).astype("float32"),
    k=1
)

print(documents[I[0][0]])

Output:

Employees receive 20 days of paid leave every year.

This is the core idea behind Retrieval-Augmented Generation.

Why RAG Is Better Than a Standalone LLM

Common Production Challenges

Building a toy RAG system is straightforward.

Building one that works well in production is much harder.

You’ll encounter challenges such as:

• Poor chunking strategies
• Weak embedding models
• Slow vector searches
• Outdated documents
• Hallucinations despite retrieval
• Duplicate chunks
• Metadata filtering
• Cost optimization
• Evaluation and monitoring

Each of these topics deserves its own deep dive—and we’ll cover them in future articles.

Where RAG Is Used

RAG powers many enterprise AI applications, including:

• Customer support assistants
• Healthcare knowledge assistants
• Insurance policy chatbots
• Legal document search
• Financial research tools
• HR assistants
• Internal company knowledge bases
• Software documentation bots

Any application that needs access to private or frequently changing information is a strong candidate for RAG.

What’s Next?

This article introduced the core concepts behind Retrieval-Augmented Generation.

In the next articles of this series, we’ll explore the building blocks that make production RAG systems effective:

1. How to Choose the Right Chunk Size for RAG
2. Embeddings Explained: How LLMs Understand Meaning
3. Vector Databases Explained: FAISS vs. Pinecone vs. Qdrant
4. Hybrid Search: Combining BM25 and Vector Search
5. Metadata Filtering for Enterprise RAG
6. How to Evaluate a RAG Pipeline
7. Production RAG Best Practices
8. Why RAG Systems Still Hallucinate—and How to Reduce It

By the end of the series, you’ll understand not only how to build a RAG system, but also how to design one that is scalable, maintainable, and ready for production.

The complete implementation used in this tutorial is available on GitHub.

⭐ If you found this tutorial helpful, consider starring the repository.

👉 GitHub Repository:https://github.com/vedprakash11/rag-fundamentals

Key Takeaways

• RAG combines information retrieval with large language models.
• It enables LLMs to answer questions using external, up-to-date knowledge.
• A typical RAG pipeline consists of document ingestion, chunking, embedding generation, vector search, prompt construction, and response generation.
• Vector databases store embeddings and enable efficient semantic search.
• Production RAG systems require careful attention to chunking, retrieval quality, metadata, evaluation, latency, and cost.

References

1. Lewis, P. et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020), NeurIPS.
2. Johnson, J., Douze, M., & Jégou, H., Billion-scale similarity search with FAISS (2019), Facebook AI Research.
3. Reimers, N., & Gurevych, I., Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019), EMNLP.
4. Vaswani, A. et al., Attention Is All You Need (2017), NeurIPS.