A practical guide to BM25, the ranking formula, TF-IDF, Python implementation, and why BM25 is still essential for modern RAG systems

Introduction

When people build a modern RAG system, the conversation often starts with embeddings:

“Which embedding model should we use?”

But before vector databases and semantic search became popular, search engines had already solved a fundamental problem:

Given a query, which documents are most relevant?

One of the most widely used solutions is BM25 — Best Matching 25.

BM25 is a lexical information-retrieval algorithm that ranks documents based primarily on term frequency, inverse document frequency, and document length.

And despite the popularity of embeddings and vector search, BM25 remains extremely useful.

Why?

Because semantic search and keyword search solve different problems.

Consider this query:

"Python pandas SettingWithCopyWarning"

A semantic retriever might find documents about:

			
DataFrame views
DataFrame copying
Pandas indexing

But a lexical retriever can strongly prioritize documents containing the exact terms:

SettingWithCopyWarning

That exact-match capability is extremely valuable for:

error messages
product names
API names
database columns
legal clauses
medical terminology
identifiers
technical documentation

This is why BM25 frequently appears alongside vector search in hybrid RAG architectures.

In this tutorial, we’ll understand exactly how BM25 works, derive its intuition from TF-IDF, implement it in Python, and see how it fits into a production RAG pipeline.

What is BM25?
Why keyword search is still important
From TF-IDF to BM25
How BM25 works
Term Frequency
Inverse Document Frequency
Document Length Normalization
The BM25 Formula
Understanding k1
Understanding b
A BM25 example
Python implementation
BM25 with a real search library
BM25 vs vector search
BM25 in RAG
Hybrid Search
Common mistakes
Production considerations
Interview questions
Key takeaways
Related tutorials
Next tutorial

What You’ll Learn

After completing this tutorial, you will understand:

What BM25 is
Why BM25 is better than simple keyword matching
How BM25 relates to TF-IDF
How term frequency affects ranking
How IDF identifies important terms
Why document length matters
What k1 and b control
How to implement BM25 in Python
BM25 vs semantic/vector search
How BM25 improves RAG retrieval
How to build hybrid retrieval using BM25 + embeddings

Prerequisites

You should have a basic understanding of:

Python
Information retrieval
TF-IDF
RAG
Embeddings

You don’t need to understand the BM25 formula beforehand.

We’ll build the intuition first.

1. What Is BM25?

BM25 stands for Best Matching 25.

It is a ranking function used in information retrieval to estimate how relevant a document is to a search query.

Given:

			
Query
  ↓
BM25
  ↓
Rank documents
  ↓
Most relevant documents

		

For example, suppose we have these documents:

			
Document 1:
"Python is a programming language used for machine learning."
Document 2:
"Python pandas provides DataFrame operations."
Document 3:
"JavaScript is commonly used for web development."

		

Query:

Python DataFrame

BM25 should rank:

			
Document 2
   ↓
Document 1
   ↓
Document 3

		

because Document 2 contains both query terms.

2. Why Not Just Count Matching Words?

You could implement a simple search system:

score = number_of_matching_words

But this creates several problems.

Suppose:

			
Query:
machine learning

Document A:

Machine learning is useful.

Document B:

			
Machine learning machine learning machine learning
machine learning machine learning...

Simply counting occurrences might heavily favor Document B.

But repeating a word 100 times doesn’t necessarily make a document 100 times more relevant.

BM25 addresses this using term-frequency saturation.

This is one of its most important ideas.

3. From TF-IDF to BM25

To understand BM25, it’s useful to start with TF-IDF.

TF-IDF combines:

			
TF = Term Frequency
IDF = Inverse Document Frequency

The intuition is simple.

Term Frequency

How often does the term appear in this document?

Inverse Document Frequency

How rare is the term across the entire collection?

For example:

Term: "the"

appears in almost every document.

It isn’t very useful for identifying a specific document.

But:

Term: "SettingWithCopyWarning"

might appear in very few documents.

It is much more informative.

BM25 builds on this intuition but introduces a better treatment of:

term-frequency saturation
document length
relevance scoring

4. How BM25 Works

BM25 essentially asks three questions:

Question 1

Does the document contain the query term?

Question 2

How frequently does the query term occur?

Question 3

How common is that term across all documents?

It also considers:

Question 4

Is the document unusually long or short?

The result is a relevance score.

                     BM25
                      │
        ┌─────────────┼─────────────┐
        ↓             ↓             ↓
       TF             IDF       Document Length
        │             │             │
        └─────────────┼─────────────┘
                      ↓
                Relevance Score

5. Term Frequency

Term Frequency measures how frequently a query term occurs inside a document.

Suppose:

			
Query:
python

Document A:

Python is popular.

Frequency:

TF = 1

Document B:

Python Python Python Python

Frequency:

TF = 4

Document B gets a higher score.

But BM25 doesn’t let the score increase indefinitely.

This is called term-frequency saturation.

The first few occurrences of a term provide useful information.

After a point, additional occurrences provide diminishing value.

Conceptually:

			
Relevance
   │
   │            ________
   │          /
   │        /
   │      /
   │    /
   │___/________________
             Term Frequency

		

This is more realistic than simply multiplying relevance by the raw number of occurrences.

6. Inverse Document Frequency

Now consider the entire document collection.

Suppose you have 1,000 documents.

The term:

python

appears in:

500 documents

while:

pandas

appears in:

100 documents

and:

SettingWithCopyWarning

appears in:

5 documents

The rare term is more useful for identifying relevant documents.

Therefore:

			
IDF(SettingWithCopyWarning)
>
IDF(pandas)
>
IDF(python)

		

This gives BM25 an important property:

Rare terms carry more discriminative power than common terms.

7. Document Length Normalization

Consider two documents.

Document A

Python pandas DataFrame

Length:

3 words

Document B

Python pandas DataFrame ...

Length:

2,000 words

Suppose both contain:

DataFrame

Should they automatically receive the same score?

Not necessarily.

A term appearing several times in a very long document may not be as informative as the same term appearing prominently in a short document.

BM25 therefore applies document-length normalization.

It compares the document length with the average document length in the collection.

8. The BM25 Formula

A commonly used form of BM25 is:

Don’t worry if this looks intimidating.

Let’s break it down.

`D`

The document.

`Q`

The query.

`t`

A query term.

`f(t,D)`

Frequency of term t in document D.

`|D|`

Length of the document.

`avgdl`

Average document length.

`k1`

Controls term-frequency saturation.

`b`

Controls document-length normalization.

9. Understanding `k1`

k1 controls how quickly term-frequency saturation happens.

A commonly used value is around:

k1 = 1.2

Higher values allow term frequency to have more influence.

Lower values cause saturation to happen earlier.

Conceptually:

			
Low k1
Term frequency
1 → big improvement
2 → smaller improvement
3 → even smaller improvement
...

		

This prevents repeated occurrences from dominating the ranking.

10. Understanding `b`

b controls how strongly BM25 normalizes document length.

Typical values are around:

b = 0.75

Interpretation:

b = 0

means little or no document-length normalization.

b = 1

means strong normalization based on document length.

In practice:

			
k1 → controls term-frequency saturation
b  → controls document-length normalization

These two parameters are among the most important BM25 tuning parameters.

11. A Simple BM25 Example

Suppose our query is:

Python DataFrame

And we have:

			
Document A:
Python DataFrame tutorial

			
Document B:
Python programming tutorial

			
Document C:
JavaScript web development

The query terms are:

			
python
dataframe

Document A contains:

			
python
dataframe

Document B contains:

python

Document C contains neither.

Therefore, we’d expect:

			
Document A
    ↓
Document B
    ↓
Document C

		

The important part is that BM25 doesn’t just ask:

“Does this document contain the word?”

It considers:

			
Term frequency
+
Term rarity
+
Document length

		

That combination produces a much better ranking.

12. Implementing BM25 in Python

Let’s first create a very small implementation to understand the mechanics.

			
import math
from collections import Counter
documents = [
    "python dataframe tutorial",
    "python programming tutorial",
    "javascript web development"
]
query = "python dataframe"

		

Tokenize the documents:

			
tokenized_docs = [
    doc.lower().split()
    for doc in documents
]
query_tokens = query.lower().split()

		

Calculate document lengths:

			
doc_lengths = [
    len(doc)
    for doc in tokenized_docs
]
avg_doc_length = sum(doc_lengths) / len(doc_lengths)
print(avg_doc_length)

		

Now calculate document frequency:

			
N = len(tokenized_docs)
df = {}
for term in query_tokens:
    df[term] = sum(
        term in doc
        for doc in tokenized_docs
    )
print(df)

		

You can then calculate the IDF component:

			
def idf(term, documents):
    N = len(documents)
    document_frequency = sum(
        term in document
        for document in documents
    )
    return math.log(
        1 + (N - document_frequency + 0.5)
        / (document_frequency + 0.5)
    )
  idf_score = idf(
            term,
            [doc.split() for doc in documents]
        )
        numerator = tf * (k1 + 1)
        denominator = (
            tf
            + k1 * (
                1 - b
                + b * document_length / avgdl
            )
        )
        score += idf_score * (
            numerator / denominator
        )
    return score

		

Now the BM25 score:

			
def bm25_score(
    query,
    document,
    documents,
    k1=1.2,
    b=0.75
):
    query_terms = query.lower().split()
    document_terms = document.lower().split()
    document_length = len(document_terms)
    avgdl = sum(
        len(doc.split())
        for doc in documents
    ) / len(documents)
    term_counts = Counter(document_terms)
    score = 0
    for term in query_terms:
        tf = term_counts[term]
        if tf == 0:
            continue
        idf_score = idf(
            term,

		

Run it:

			
scores = [
    bm25_score(
        query,
        document,
        documents
    )
    for document in documents
]
for document, score in zip(documents, scores):
    print(score, document)

		

You’ll see the document containing both:

			
python
dataframe

rank above the others.

This implementation is intentionally simple.

In production, you normally use a tested information-retrieval implementation rather than maintaining your own BM25 implementation.

13. Using BM25 in a Real Python Application

One popular approach is to use a library implementation rather than calculating BM25 manually.

For example:

			
from rank_bm25 import BM25Okapi
documents = [
    "Python dataframe tutorial",
    "Python programming tutorial",
    "JavaScript web development"
]
tokenized_docs = [
    doc.lower().split()
    for doc in documents
]
bm25 = BM25Okapi(tokenized_docs)
query = "python dataframe"
tokenized_query = query.lower().split()
scores = bm25.get_scores(tokenized_query)
print(scores)

		

You can retrieve the top documents:

			
top_documents = bm25.get_top_n(
    tokenized_query,
    documents,
    n=2
)
print(top_documents)

		

The key idea is:

			
Documents
    ↓
Tokenization
    ↓
BM25 Index
    ↓
Query
    ↓
BM25 Scores
    ↓
Top-K Documents

		

14. BM25 vs Vector Search

This is where BM25 becomes particularly interesting for modern AI systems.

Suppose the user searches:

"SettingWithCopyWarning"

BM25 is very good at exact lexical matching.

Vector search instead tries to understand semantic similarity.

For example:

			
Query:
"How do I avoid modifying a pandas DataFrame view?"

A vector retriever might find:

"How pandas DataFrame views and copies work"

even though the exact words don’t match.

So the two approaches have different strengths.

Feature	BM25	Vector Search
Exact keywords	Excellent	Variable
Semantic similarity	Limited	Excellent
Rare technical terms	Excellent	Can vary
Typos	Limited	Can help
Synonyms	Limited	Strong
Identifiers	Excellent	Can vary
No labeled training data	Yes	Usually embeddings required
RAG use	Excellent	Excellent

This leads to an important conclusion:

BM25 and vector search are not necessarily competitors. They can complement each other.

15. Why BM25 Is Important for RAG

A typical RAG system looks like:

			
User Query
     │
     ↓
Retriever
     │
     ├──────────────┐
     ↓              ↓
Vector Search     BM25
     │              │
     └──────┬───────┘
            ↓
       Merge Results
            ↓
         Reranker
            ↓
      Top-K Context
            ↓
           LLM
            ↓
        Final Answer

		

This is called hybrid retrieval.

Imagine the user asks:

“What does Article 17.3(b) of the policy say?”

A vector retriever may understand the semantic meaning.

But BM25 is excellent at identifying:

Article 17.3(b)

The exact identifier matters.

This is common in:

legal documents
insurance policies
technical documentation
financial reports
software documentation
medical literature

16. Hybrid Search: BM25 + Vector Search

A simple hybrid approach can combine normalized scores:

			
final_score = (
    alpha * vector_score
    + (1 - alpha) * bm25_score
)

For example:

			
alpha = 0.7
final_score = (
    0.7 * vector_score
    + 0.3 * bm25_score
)

		

The exact weighting should not simply be guessed.

Tune it against your evaluation dataset.

An even better production architecture can use:

			
BM25
 +
Dense Retrieval
      ↓
Candidate Pool
      ↓
Cross Encoder
      ↓
Top-K Context

		

This gives you:

Lexical matching

Semantic matching

Deep relevance scoring

17. When Does BM25 Beat Vector Search?

BM25 can be particularly strong when queries contain exact terms.

Example 1 — Error Messages

"CUDA out of memory"

Exact lexical matching is extremely useful.

Example 2 — API Names

"read_parquet"

You want documents containing the exact function.

Example 3 — Product IDs

"SKU-84921"

Semantic similarity isn’t necessarily useful.

Example 4 — Legal Clauses

"Section 12.4(a)"

Exact matching can be critical.

Example 5 — Technical Terms

"SettingWithCopyWarning"

A search system should strongly favor documents containing the exact term.

18. When Vector Search Is Better

Vector search becomes especially useful when the query and document use different words.

Query:

"How can I make my model cheaper to run?"

Document:

"Reducing inference costs through quantization"

There may be limited exact word overlap.

But the concepts are strongly related.

Semantic search can bridge that gap.

This is why production RAG systems increasingly use both.

19. Common BM25 Mistakes

Mistake 1: Treating BM25 as Semantic Search

BM25 doesn’t understand meaning in the same way an embedding model does.

These may not match strongly:

"automobile"

and:

"car"

unless the documents contain overlapping terms or additional processing helps.

Mistake 2: Ignoring Tokenization

Your tokenizer directly affects BM25.

For example:

"read_parquet"

versus:

"read parquet"

can produce different retrieval behavior.

For technical search, tokenization should preserve important identifiers where appropriate.

Mistake 3: Using Too Many Retrieved Documents

BM25 might retrieve many keyword-matching documents.

Don’t blindly send all of them to the LLM.

Use:

			
BM25
 ↓
Top 20
 ↓
Reranker
 ↓
Top 5
 ↓
LLM

		

instead of:

			
BM25
 ↓
Top 100
 ↓
LLM

		

Mistake 4: Never Evaluating Retrieval

Don’t assume BM25 is working because the search results “look reasonable.”

Measure:

			
Precision@K
Recall@K
MRR
nDCG

and ultimately:

			
Answer correctness
Groundedness
Task success

20. Production Considerations

If you’re implementing BM25 in a production RAG system, pay attention to the following.

1. Tokenization

Use consistent preprocessing for:

			
Documents
Queries

2. Stop Words

Depending on your use case, stop-word handling can affect retrieval.

Don’t blindly remove terms from technical or domain-specific queries.

3. Stemming/Lemmatization

Depending on the domain, you may benefit from normalization such as:

			
connect
connected
connecting

But aggressive normalization can hurt technical searches.

Test it.

4. Index Updates

Your BM25 index needs to reflect document changes.

For dynamic knowledge bases, consider how frequently the index should be rebuilt or updated.

5. Chunking

BM25 works on the text you give it.

Poor chunking can therefore produce poor retrieval.

For RAG:

			
Documents
    ↓
Chunking
    ↓
BM25 Index

		

Chunk size matters.

6. Evaluation

Maintain a retrieval evaluation dataset:

			
Query
Expected Documents
Relevant Documents

Then measure retrieval quality whenever you change:

tokenization
chunk size
preprocessing
BM25 parameters
hybrid-search weights

21. BM25 Interview Questions

If you’re preparing for an AI Engineer or ML Engineer interview, expect questions like these.

Beginner

1. What is BM25?

BM25 is a lexical information-retrieval ranking algorithm used to rank documents according to their relevance to a query.

2. What does BM25 stand for?

Best Matching 25.

3. What are the main components of BM25?

Term frequency, inverse document frequency, and document-length normalization.

Intermediate

4. How is BM25 different from TF-IDF?

BM25 extends the basic TF-IDF intuition by introducing term-frequency saturation and document-length normalization.

5. What does k1 control?

Term-frequency saturation.

6. What does b control?

Document-length normalization.

7. Why doesn’t BM25 simply count keyword occurrences?

Because repeated occurrences should have diminishing returns.

Advanced

8. Why use BM25 with vector search?

BM25 provides strong lexical matching while vector search provides semantic matching.

9. When can BM25 outperform vector search?

Queries involving exact technical terms, identifiers, product names, error messages, or specific clauses can benefit significantly from lexical matching.

10. How would you evaluate a BM25 retriever?

Using metrics such as:

			
Precision@K
Recall@K
MRR
nDCG

and ultimately measuring downstream RAG metrics such as groundedness and answer correctness.

Key Takeaways

BM25 isn’t a new deep-learning architecture.

It doesn’t use neural networks.

It doesn’t generate embeddings.

And it doesn’t understand language semantically in the way modern embedding models do.

Yet it remains extremely useful.

The core idea is surprisingly simple:

                    BM25
                      │
          ┌───────────┼───────────┐
          ↓           ↓           ↓
         TF          IDF       Doc Length
          │           │           │
          └───────────┼───────────┘
                      ↓
                Ranking Score

Remember these four concepts:

1. Term Frequency

How frequently does the query term appear?

2. Inverse Document Frequency

How rare is the term across the collection?

3. Term-Frequency Saturation

Repeated occurrences provide diminishing returns.

4. Document-Length Normalization

Long documents shouldn’t automatically win simply because they contain more words.

And in modern RAG:

			
BM25
+
Vector Search
+
Reranking
=
Powerful Hybrid Retrieval

		

The biggest lesson is not that BM25 replaces vector search.

It’s that different retrieval strategies capture different signals.

A production RAG system should choose the retrieval architecture based on the nature of its data and queries—not simply because vector databases are popular.

Next Tutorial

← Previous

Dense vs Sparse Retrieval: What’s the Difference?

You are here:

RAG Fundamentals → BM25 Explained

Next →

Hybrid Search: Combining BM25 and Vector Search

Author

Ved Prakash

Senior Data Scientist | AI Engineer | Generative AI( LinkedIn)

Writing practical tutorials on Generative AI, RAG, LLMs, AI Agents, Data Engineering, Machine Learning, and production AI systems.

A practical guide to BM25, the ranking formula, TF-IDF, Python implementation, and why BM25 is still essential for modern RAG systems

Introduction

Table of Contents

What You’ll Learn

Prerequisites

1. What Is BM25?

2. Why Not Just Count Matching Words?

3. From TF-IDF to BM25

Term Frequency

Inverse Document Frequency

4. How BM25 Works

Question 1

Question 2

Question 3

Question 4

5. Term Frequency

6. Inverse Document Frequency

7. Document Length Normalization

Document A

Document B

8. The BM25 Formula

D

Q

t

f(t,D)

|D|

avgdl

k1

b

9. Understanding k1

10. Understanding b

11. A Simple BM25 Example

12. Implementing BM25 in Python

13. Using BM25 in a Real Python Application

14. BM25 vs Vector Search

15. Why BM25 Is Important for RAG

16. Hybrid Search: BM25 + Vector Search

17. When Does BM25 Beat Vector Search?

Example 1 — Error Messages

Example 2 — API Names

Example 3 — Product IDs

Example 4 — Legal Clauses

Example 5 — Technical Terms

18. When Vector Search Is Better

19. Common BM25 Mistakes

Mistake 1: Treating BM25 as Semantic Search

Mistake 2: Ignoring Tokenization

Mistake 3: Using Too Many Retrieved Documents

Mistake 4: Never Evaluating Retrieval

20. Production Considerations

1. Tokenization

2. Stop Words

3. Stemming/Lemmatization

4. Index Updates

5. Chunking

6. Evaluation

21. BM25 Interview Questions

Beginner

Intermediate

Advanced

Key Takeaways

1. Term Frequency

2. Inverse Document Frequency

3. Term-Frequency Saturation

4. Document-Length Normalization

Related Tutorials

RAG Fundamentals

Advanced RAG

Next Tutorial

Author

Share this:

Like this:

Related

Leave a ReplyCancel reply

Discover more from Geeky Codes

`D`

`Q`

`t`

`f(t,D)`

`|D|`

`avgdl`

`k1`

`b`

9. Understanding `k1`

10. Understanding `b`