A practical guide to lexical + semantic retrieval, score fusion, reranking, Python implementation, and production RAG architecture

Introduction

A common mistake when building a RAG system is assuming that vector search is enough.

You convert documents into embeddings, store them in a vector database, retrieve the top-k chunks, and send them to the LLM.

It works.

Until a user asks:

“What is the error SettingWithCopyWarning?”

or:

“What does Section 12.4(b) say?”

or:

“How do I use pd.read_parquet()?”

These queries contain exact terms that matter.

A semantic retriever may understand the general meaning of the query, but a lexical retriever such as BM25 can be much better at matching exact keywords, identifiers, error messages, and technical terminology.

On the other hand, consider:

“How can I make my LLM application cheaper?”

A document might say:

“Reducing inference expenditure through quantization and model routing.”

There may be little exact word overlap, but the meanings are closely related.

This is where hybrid search becomes useful.

Instead of asking:

BM25 or vector search?

we combine:

                 User Query
                     │
          ┌──────────┴──────────┐
          ↓                     ↓
     BM25 Search          Vector Search
          │                     │
          ↓                     ↓
    Keyword Results       Semantic Results
          │                     │
          └──────────┬──────────┘
                     ↓
                Score Fusion
                     ↓
                  Reranker
                     ↓
               Top-K Chunks
                     ↓
                    LLM
                     ↓
                Final Answer

The result is a retrieval system that can capture both lexical relevance and semantic relevance.

In this tutorial, we’ll build hybrid search from first principles and then discuss how to use it in production RAG systems.

Table of Contents

1. What Is Hybrid Search?
2. Why Vector Search Alone Isn’t Enough
3. Why BM25 Alone Isn’t Enough
4. BM25 vs Vector Search
5. How Hybrid Search Works
6. Hybrid Search Architecture
7. Step 1 — Prepare Documents
8. Step 2 — Implement BM25 Retrieval
9. Step 3 — Implement Vector Retrieval
10. Step 4 — Normalize Scores
11. Step 5 — Combine Results
12. Complete Python Example
13. Reciprocal Rank Fusion
14. Hybrid Search with Reranking
15. Hybrid Search in RAG
16. When Hybrid Search Works Best
17. Common Mistakes
18. Production Considerations
19. How to Evaluate Hybrid Search
20. Interview Questions
21. Key Takeaways
22. Related Tutorials
23. Next Tutorial

What You’ll Learn

By the end of this tutorial, you’ll understand:

• What hybrid search is
• Why lexical and semantic retrieval complement each other
• How BM25 works inside a hybrid retriever
• How vector search contributes semantic matching
• How to combine retrieval results
• Score normalization
• Weighted score fusion
• Reciprocal Rank Fusion
• Reranking
• How hybrid retrieval fits into RAG
• How to evaluate hybrid retrieval
• Production considerations

Prerequisites

You should be familiar with:

• Python
• Basic information retrieval
• BM25
• Embeddings
• Vector databases
• RAG

If you’re new to BM25, read:

BM25 Explained: How Search Engines Find the Most Relevant Documents

before continuing.

1. What Is Hybrid Search?

Hybrid search combines multiple retrieval strategies to retrieve relevant documents.

The most common combination is:

BM25
+
Vector Search

BM25 performs lexical retrieval.

Vector search performs semantic retrieval.

So instead of relying on one signal:

Query
 ↓
Vector Search
 ↓
Documents

we use:

                 Query
                   │
          ┌────────┴────────┐
          ↓                 ↓
        BM25             Embeddings
          ↓                 ↓
     Keyword Search    Semantic Search
          │                 │
          └────────┬────────┘
                   ↓
              Combination
                   ↓
              Final Ranking

This is particularly useful for RAG systems because enterprise queries often contain a mixture of:

• natural language
• exact keywords
• technical terminology
• identifiers
• product names
• document sections
• error messages

2. Why Vector Search Alone Isn’t Enough

Vector search works by converting text into numerical representations called embeddings.

For example:

"How can I reduce LLM inference costs?"

might become:

[0.12, -0.45, 0.87, ...]

A document is converted into another vector.

Then we calculate similarity, commonly using:

Cosine Similarity

Conceptually:

Query Embedding
      │
      ↓
Vector Database
      │
      ↓
Nearest Neighbors
      │
      ↓
Relevant Documents

This works very well for semantic queries.

For example:

Query:
How can I reduce the cost of running an LLM?
Document:
Techniques for reducing inference expenditure include
quantization, caching, batching and model routing.

The words aren’t identical, but the meanings are related.

However, vector search can struggle with exact identifiers.

Consider:

"CUDA error 718"

or:

"EC2 i4i.2xlarge"

or:

"Article 12.4(b)"

or:

"SettingWithCopyWarning"

These are cases where exact lexical matching is valuable.

3. Why BM25 Alone Isn’t Enough

BM25 has the opposite limitation.

It is excellent at matching words.

But it doesn’t inherently understand semantic similarity.

Consider:

Query:
How do I make inference cheaper?

Document:

Quantization reduces the computational and memory
requirements of model serving.

The terms don’t overlap much.

BM25 may rank this document lower than it should.

Vector search can recognize that:

cheaper inference

and:

reduced model serving cost

are semantically related.

So:

BM25
→ exact matching
Vector Search
→ semantic matching

This is the fundamental motivation behind hybrid search.

4. BM25 vs Vector Search

Neither approach is universally better.

They retrieve using different signals.

5. How Hybrid Search Works

Suppose the user asks:

How do I fix CUDA out of memory during PyTorch training?

BM25 might return:

1. CUDA Out of Memory Errors
2. PyTorch GPU Memory Management
3. CUDA Memory Troubleshooting

Vector search might return:

1. Reducing GPU memory consumption during training
2. Gradient checkpointing
3. Optimizing batch size

Notice that both systems found useful information, but for different reasons.

We can merge these candidates:

BM25 Results
       │
       ├── Document A
       ├── Document B
       └── Document C
       
Vector Results
       │
       ├── Document D
       ├── Document E
       └── Document F
       
       ↓
   
Candidate Pool
   
       ↓
   
Final Ranking

6. Hybrid Search Architecture

A production RAG pipeline can look like this:

                         User Query
                              │
                              ▼
                    Query Preprocessing
                              │
                 ┌────────────┴────────────┐
                 │                         │
                 ▼                         ▼
             BM25 Index              Embedding Model
                 │                         │
                 ▼                         ▼
          Lexical Retrieval         Vector Retrieval
                 │                         │
                 └────────────┬────────────┘
                              ▼
                       Result Fusion
                              │
                              ▼
                         Top-N Candidates
                              │
                              ▼
                         Cross Encoder
                           Reranker
                              │
                              ▼
                          Top-K Chunks
                              │
                              ▼
                             LLM
                              │
                              ▼
                         Final Answer

This architecture separates retrieval into multiple stages.

7. Step 1 — Prepare Documents

Let’s start with a small collection.

documents = [
    "Python pandas DataFrame tutorial",
    "How to fix SettingWithCopyWarning in pandas",
    "PyTorch CUDA out of memory troubleshooting",
    "Reducing LLM inference costs with quantization",
    "Building RAG systems with vector databases",
    "Hybrid search using BM25 and embeddings"
]

In a real RAG application, these would typically be chunks generated from:

PDFs
Web pages
Documentation
Database records
Internal knowledge bases

8. Step 2 — Implement BM25 Retrieval

Install the library:

pip install rank-bm25

Tokenize the documents:

from rank_bm25 import BM25Okapi
tokenized_documents = [
    document.lower().split()
    for document in documents
]
bm25 = BM25Okapi(tokenized_documents)

Now define a query:

query = "pandas SettingWithCopyWarning"
tokenized_query = query.lower().split()

Retrieve documents:

scores = bm25.get_scores(tokenized_query)

Rank them:

ranked_documents = sorted(
    zip(documents, scores),
    key=lambda x: x[1],
    reverse=True
)
for document, score in ranked_documents:
    print(score, document)

BM25 will strongly favor documents containing the exact query terms.

9. Step 3 — Implement Vector Retrieval

Now let’s add semantic search.

For a simple demonstration, we can use Sentence Transformers.

Install:

pip install sentence-transformers

Load an embedding model:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
    "all-MiniLM-L6-v2"
)

Generate document embeddings:

document_embeddings = model.encode(
    documents,
    normalize_embeddings=True
)

Generate the query embedding:

query_embedding = model.encode(
    query,
    normalize_embeddings=True
)

Calculate cosine similarity:

import numpy as np
similarities = (
    document_embeddings @ query_embedding
)

Rank the results:

vector_results = sorted(
    zip(documents, similarities),
    key=lambda x: x[1],
    reverse=True
)
for document, score in vector_results:
    print(score, document)

Now we have two independent retrieval systems:

BM25
 ↓
Lexical scores
Vector Search
 ↓
Semantic scores

10. Step 4 — Normalize Scores

Here’s an important problem.

BM25 scores and vector similarity scores are on different scales.

For example:

BM25:
Document A → 8.4
Document B → 4.2
Document C → 1.7

Vector similarity:

Document A → 0.82
Document B → 0.71
Document C → 0.43

You can’t blindly add them:

final_score = bm25_score + vector_score

because the scales aren’t comparable.

One simple approach is min-max normalization.

def normalize(scores):
    scores = np.array(scores)
    min_score = scores.min()
    max_score = scores.max()
    if max_score == min_score:
        return np.zeros_like(scores)
    return (
        (scores - min_score)
        / (max_score - min_score)
    )

Apply it:

bm25_scores = normalize(

[score for _, score in ranked_documents]

) vector_scores = normalize(

[score for _, score in vector_results]

)

Now both scores are approximately:

0 → 1

11. Step 5 — Combine Results

A simple weighted fusion approach is:

For example:

alpha = 0.7

means:

70% semantic relevance
30% lexical relevance

Implementation:

hybrid_score = (
    alpha * vector_score
    +
    (1 - alpha) * bm25_score
)

For example:

alpha = 0.7
final_score = (
    0.7 * vector_score
    + 0.3 * bm25_score
)

However, this isn’t the only way to combine rankings.

And in many systems, Reciprocal Rank Fusion (RRF) is a better starting point.

12. Reciprocal Rank Fusion

Instead of combining raw scores, we can combine rankings.

This is called:

Reciprocal Rank Fusion (RRF).

The formula is:

where:

• d = document
• r = retrieval system
• rank = document’s rank
• k = constant

The idea is simple.

Suppose:

BM25

1. Document A
2. Document B
3. Document C

Vector Search

1. Document C
2. Document A
3. Document D

RRF rewards documents that consistently appear near the top.

For example:

Document A
BM25 rank = 1
Vector rank = 2
Document C
BM25 rank = 3
Vector rank = 1

Both are strong candidates.

13. Python RRF Implementation

def reciprocal_rank_fusion(
    rankings,
    k=60
):
    scores = {}
    for ranking in rankings:
        for rank, document in enumerate(
            ranking,
            start=1
        ):
            scores[document] = (
                scores.get(document, 0)
                + 1 / (k + rank)
            )
    return sorted(
        scores.items(),
        key=lambda x: x[1],
        reverse=True
    )

Now we can create separate rankings:

bm25_ranking = [
    "Document A",
    "Document B",
    "Document C"
]
vector_ranking = [
    "Document C",
    "Document A",
    "Document D"
]

Fuse them:

result = reciprocal_rank_fusion(
    [
        bm25_ranking,
        vector_ranking
    ]
)
print(result)

RRF is attractive because it avoids the problem of comparing incompatible score scales.

14. Hybrid Search With Reranking

Retrieval doesn’t necessarily have to end after score fusion.

A stronger architecture is:

                 Query
                   │
         ┌─────────┴─────────┐
         ↓                   ↓
       BM25              Vector Search
         │                   │
         └─────────┬─────────┘
                   ↓
             Candidate Pool
                   │
                   ↓
             Cross Encoder
                   │
                   ↓
                Top-K
                   │
                   ↓
                  LLM

For example:

BM25 → Top 20
Vector → Top 20
         ↓
      Merge
         ↓
    30 unique chunks
         ↓
    Reranker
         ↓
     Top 5
         ↓
       LLM

Why do this?

Because retrieval and reranking solve different problems.

Retrieval

Fast candidate generation.

Reranking

More expensive relevance estimation.

This gives us:

High recall
+
High precision

15. Hybrid Search in RAG

Now let’s connect everything to RAG.

A complete architecture might be:

                         User
                           │
                           ▼
                         Query
                           │
             ┌─────────────┴─────────────┐
             │                           │
             ▼                           ▼
        BM25 Retriever            Vector Retriever
             │                           │
             │                           │
             └─────────────┬─────────────┘
                           ▼
                      Result Fusion
                           │
                           ▼
                    Candidate Documents
                           │
                           ▼
                       Reranker
                           │
                           ▼
                      Top-K Chunks
                           │
                           ▼
                    Prompt Construction
                           │
                           ▼
                          LLM
                           │
                           ▼
                    Grounded Answer

This approach is especially useful for enterprise RAG.

For example, imagine an insurance knowledge base.

User asks:

“What does policy clause 7.2 say about accidental hospitalization?”

BM25 can identify:

Clause 7.2

while vector search can identify documents discussing:

accidental hospitalization

Combining both signals increases the chance that the correct chunk reaches the LLM.

16. When Hybrid Search Works Best

Hybrid retrieval is particularly useful for datasets containing:

Technical Documentation

"read_parquet"
"SparkSession"
"SettingWithCopyWarning"
"CUDA out of memory"

Legal Documents

"Section 12.4(b)"
"Article 17"
"Clause 8.2"

Financial Documents

"ISIN INE..."
"Form 10-K"
"EBITDA"

Healthcare

"ICD-10"
"HCPCS"
"drug identifiers"

E-commerce

"SKU-12345"
"iPhone 17 Pro"

The more important exact terminology is, the more valuable lexical retrieval becomes.

17. When You May Not Need Hybrid Search

Hybrid search isn’t automatically the right solution.

If your dataset consists mostly of:

Natural-language questions
+
Natural-language documents

and exact identifiers aren’t important, pure semantic retrieval may perform well.

Similarly, adding another retrieval system increases:

• infrastructure complexity
• latency
• operational overhead
• evaluation complexity

Don’t add BM25 simply because “production RAG uses hybrid search.”

Benchmark it.

18. Common Mistakes

Mistake 1: Combining Raw Scores

Avoid:

final_score = bm25_score + vector_score

unless you’ve deliberately calibrated the scores.

They may have completely different distributions.

Use:

• normalized scores
• rank fusion
• RRF
• learned fusion

instead.

Mistake 2: Assuming 50/50 Is Optimal

You might see:

alpha = 0.5

and assume this is correct.

It isn’t necessarily.

For some datasets:

BM25 → 70%
Vector → 30%

may perform better.

For others:

BM25 → 20%
Vector → 80%

may be better.

Use an evaluation dataset to determine this.

Mistake 3: Retrieving Too Few Candidates

Suppose:

BM25 → Top 3
Vector → Top 3

You may miss relevant documents.

A common architecture is:

BM25 → Top 20
Vector → Top 20
      ↓
Merge
      ↓
Rerank
      ↓
Top 5

The exact numbers should depend on latency, corpus size, and evaluation results.

Mistake 4: Ignoring Duplicate Documents

Both retrievers may return the same chunk.

Always deduplicate before reranking.

unique_documents = list(
    dict.fromkeys(
        bm25_results + vector_results
    )
)

Mistake 5: Ignoring Chunking

Hybrid retrieval cannot fix poor chunking.

If your chunks are too large:

Large Chunk
 ├── Topic A
 ├── Topic B
 ├── Topic C
 └── Topic D

the retrieved context may be noisy.

If they’re too small:

Chunk A → incomplete sentence
Chunk B → missing context
Chunk C → isolated fact

retrieval may lose important information.

Chunking should therefore be evaluated together with retrieval.

19. Production Considerations

A production hybrid RAG system should consider more than retrieval quality.

Latency

Running two retrieval systems means:

BM25
+
Vector Search

can introduce additional latency.

Run them in parallel when possible:

              Query
                │
        ┌───────┴───────┐
        ↓               ↓
      BM25           Vector DB
        │               │
        └───────┬───────┘
                ↓
             Fusion

Caching

Cache repeated queries.

For example:

Query
 ↓
Hash
 ↓
Cache
 ↓
Retrieved results

This can reduce repeated retrieval work.

Indexing

Maintain:

BM25 Index
+
Vector Index

when documents are created, updated, or deleted.

Your two indexes must remain reasonably synchronized.

Metadata Filtering

Don’t rely exclusively on retrieval ranking.

Apply metadata filters where appropriate:

document_type = "policy"
country = "US"
version = "2026"

A good pipeline might be:

Metadata Filter
      ↓
BM25 + Vector Search
      ↓
Fusion
      ↓
Reranking

This reduces irrelevant candidates before the expensive stages.

20. How to Evaluate Hybrid Search

This is one of the most important parts of a production RAG system.

Don’t evaluate only the final LLM answer.

Evaluate retrieval separately.

Create a golden dataset:

Query
Expected Document IDs
Relevant Chunks

For example:

Then measure:

Recall@K

How many relevant documents were retrieved?

Precision@K

How many retrieved documents were actually relevant?

MRR

How high did the first relevant result appear?

nDCG

How good was the overall ranking?

Then evaluate the downstream RAG system:

Retrieval
   ↓
Context Relevance
   ↓
Context Recall
   ↓
Faithfulness
   ↓
Answer Correctness
   ↓
Task Success

The goal isn’t simply:

“Did retrieval improve?”

The real question is:

Did better retrieval improve the user’s outcome?

21. Hybrid Search Interview Questions

1. What is hybrid search?

Hybrid search combines multiple retrieval strategies, commonly BM25 lexical retrieval and vector semantic retrieval.

2. Why combine BM25 and vector search?

Because BM25 is strong at exact lexical matching while vector search is strong at semantic similarity.

3. When would BM25 outperform vector search?

For exact terms such as:

• error messages
• product IDs
• API names
• legal clauses
• technical identifiers

4. How do you combine BM25 and vector scores?

Possible approaches include:

• score normalization + weighted fusion
• Reciprocal Rank Fusion
• learned ranking models

5. Why use RRF?

RRF combines rankings rather than raw scores, avoiding the need to calibrate different score distributions.

6. Where would you put a reranker?

After candidate retrieval and fusion:

BM25 + Vector
      ↓
Candidate Pool
      ↓
Reranker
      ↓
Top-K

7. How would you evaluate a hybrid retriever?

Use:

Recall@K
Precision@K
MRR
nDCG

and evaluate downstream RAG quality.

8. How would you decide between BM25, vector search, and hybrid search?

Benchmark them against representative production queries.

Don’t choose based purely on architecture trends.

Key Takeaways

Hybrid search isn’t about choosing between traditional information retrieval and modern AI.

It’s about combining their strengths.

                 Hybrid Retrieval
                       │
             ┌─────────┴─────────┐
             ↓                   ↓
           BM25              Vector Search
             │                   │
             ↓                   ↓
      Exact Matching       Semantic Matching
             │                   │
             └─────────┬─────────┘
                       ↓
                  Fusion / RRF
                       ↓
                    Reranker
                       ↓
                    Top-K
                       ↓
                      LLM

Remember:

BM25 is strong at exact lexical matching.

Vector search is strong at semantic similarity.

Hybrid search combines both.

Reranking can further improve precision.

And evaluation determines whether the architecture actually works.

The most important production principle is:

Don’t optimize your retrieval architecture based on what is popular. Optimize it based on the queries your users actually ask.

Related Tutorials

RAG Fundamentals

• What Is RAG?
• How RAG Works
• Why LLMs Hallucinate
• Dense vs Sparse Retrieval
• BM25 Explained
• Hybrid Search: BM25 + Vector Search(You are here)
• Cross-Encoder Reranking
• Parent-Child Retrieval
• Multi-Query Retrieval

Advanced RAG

10. Self-RAG
11. Corrective RAG
12. Graph RAG
13. Agentic RAG
14. RAG Evaluation
15. Production RAG Architecture

Next Tutorial

← Previous

BM25 Explained: How Search Engines Find the Most Relevant Documents

You are here:

RAG Fundamentals → Hybrid Search

Next →

Cross-Encoder Reranking: How to Improve RAG Retrieval

Author

Ved Prakash

Senior Data Scientist | AI Engineer | Generative AI

Writing practical tutorials on RAG, LLMs, AI Agents, LangGraph, MCP, Machine Learning, and production AI systems.