<aside> 📘

Agument Code Series：

How we made code search 40% faster for 100M+ line codebases using quantized vector search - Augment Code

How to build your Agent: 11 prompting techniques for better AI agents - Augment Code

📒 Integrate BM25:

🤖 Combining Embeddings with BM25 for Chunk Indexing

PostgreSQL BM25 Full-Text Search: Speed Up Performance with These Tips

📒 ANN

🤖 Implementing ANN-Based Semantic Search with Go and PostgreSQL

Optimize generative AI applications with pgvector indexing: A deep dive into IVFFlat and HNSW techniques | AWS Database Blog

https://github.com/pgvector/pgvector

📒 RAG Related:

https://blog.laisky.com/p/llm-rag/

Contextual Retrieval in AI Systems \ Anthropic

</aside>

Combining Embeddings with BM25 for Chunk Indexing: Concepts, Implementations, and Hybrid Strategies

Introduction

Retrieval-Augmented Generation (RAG) systems, designed to augment large language models (LLMs) by injecting relevant external knowledge, stand or fall on their information retrieval component. Recent advances have seen the rise of hybrid retrieval strategies that combine dense vector embeddings and classical lexical search, especially with BM25. Anthropic’s "Contextual Retrieval" methodology highlights how mixing dense and sparse retrieval over intelligently chunked, contextualized text can dramatically improve retrieval success rates—reducing failures by up to 67% when paired with reranking models. This report presents a thorough, research-driven exploration of BM25’s theoretical and practical foundation, its integration in LLM-oriented pipelines, hybrid strategies for marrying embeddings and keyword scoring, best practices around chunking and indexing, and actionable implementation examples in Go, including a minimal working hybrid retriever.

I. BM25: Mathematical Foundations and Contemporary Relevance

1. Origins and Motivation

BM25 (Best Matching 25) is a probabilistic ranking function rooted in the probabilistic information retrieval framework developed over several decades by Robertson, Spärck Jones, and colleagues. Born as the default scoring function in the Okapi Information Retrieval System, BM25 improved on TF-IDF by refining the treatment of term rarity, saturation effects in term frequency, and length normalization—in effect blending statistical rigor with practical adaptability.

BM25 has been pivotal in both academic and industrial search, powering anything from small-scale document retrieval to the backbone ranking in engines like Elasticsearch, Apache Lucene, and OpenSearch.

2. The BM25 Scoring Formula, Step by Step

The BM25 score for a document D with respect to a query Q composed of terms q₁, ..., qₙ is:

$$ Score(D,Q)=∑i=1nIDF(qi)⋅f(qi,D)⋅(k1+1)f(qi,D)+k1⋅(1−b+b⋅∣D∣avgdl)\mathrm{Score}(D, Q) = \sum_{i=1}^n IDF(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot (1 - b + b \cdot \frac{|D|}{\mathrm{avgdl}})} $$

• f(qᵢ, D): frequency of term qᵢ in document D.
• |D|: length of document D, usually in tokens or words.
• avgdl: average document length in the corpus.
• k₁: term frequency saturation parameter (typ. 1.2 – 2.0); controls how quickly additional occurrences of a term lose impact.
• b: length normalization parameter (typ. 0.75); 0 means no normalization, 1 means full normalization.
• IDF(qᵢ): inverse document frequency, rewarding rare terms:

$$ IDF(qi)=ln⁡(N−n(qi)+0.5n(qi)+0.5+1)IDF(q_i) = \ln \left( \frac{N - n(q_i) + 0.5}{n(q_i) + 0.5} + 1 \right) $$

Where:

• N: total number of documents.