Learn · Beginner

BM25 and lexical search: the keyword formula that keeps beating neural retrieval

BM25 is a formula for deciding which documents best match a set of query words. It scores a document higher when the query's rarer words appear in it often, and lower when the document is long enough that those appearances could be coincidence. It was designed in the 1990s, involves no neural network and no training, and it remains the baseline that modern AI retrieval systems are measured against - a baseline they frequently fail to beat.

That is a strange thing to say in 2026, so it is worth understanding why it keeps happening.

The problem retrieval solves

When an AI system answers a question using documents you supply - the pattern known as retrieval-augmented generation - something has to decide which documents to hand it. A model can only read a limited amount of text, so out of a million documents, roughly five will actually make it into the prompt. The quality of the answer is capped by whether those five were the right five. Retrieval is that decision, and it happens before the model does any thinking at all.

How BM25 works

Imagine searching a company wiki for "quarterly revenue Estonia". A naive approach counts how many of those three words each document contains. That immediately breaks: every document contains "revenue", so counting it tells you nothing, while "Estonia" appears in four documents and is enormously informative.

BM25 encodes three corrections to naive counting, and each one is a piece of common sense made arithmetic.

First, rare words matter more. A term appearing in five documents out of a million carries far more signal than one appearing in half of them. BM25 weights each query word by how rare it is across the whole collection - the idea called inverse document frequency, which predates BM25 by decades.

Second, repetition saturates. A document mentioning "Estonia" twenty times is more relevant than one mentioning it once, but it is not twenty times more relevant. BM25 applies a curve that rises quickly at first and then flattens, so no single word can dominate the score by sheer repetition. This is the correction that keeps keyword stuffing from working.

Third, length is discounted. A 400-page document contains more of every word by accident. BM25 normalises against the average document length in the collection, so a short document that mentions Estonia twice outranks a book that mentions it three times.

That is the entire method. Two tunable knobs control how aggressively the saturation and length corrections apply, and the standard defaults have worked well enough for thirty years that most people never touch them.

Why it refuses to lose

Neural retrieval works differently. It converts documents and queries into embeddings - lists of numbers positioned so that things with similar meaning sit near each other - and finds the documents nearest to the query. This handles the case BM25 genuinely cannot: a query about "car" matching a document about "automobile", with no shared words.

But BM25 has two structural advantages that survive every generation of neural models. It cannot be out of domain, because it was never trained on any domain; drop it on medical records, legal filings or a codebase and it behaves identically. And exact matching is not a primitive relic - it is precisely what you want when the query contains an error code, a part number, a surname or a gene. Embedding models routinely retrieve something semantically similar to a rare identifier, which is the wrong answer delivered confidently.

Jimmy Lin's 2019 paper The Neural Hype and Comparisons Against Weak Baselines documented the recurring failure: a large fraction of published neural retrieval improvements were measured against a poorly tuned BM25, and evaporated against a properly tuned one. That critique still bites.

The most recent evidence arrived in July 2026. A controlled scaling study grew the same corpus across 28 nested sizes and compared BM25, dense retrieval, graph indexing and an AI agent that browsed files. The agent won at the smallest sizes while spending 39 times more query tokens - then BM25 overtook it around 10 million corpus tokens and led at every larger size by a margin approaching 20 points. The authors' conclusion: "agentic reasoning works best after ranked discovery rather than in place of it."

What to do with this

Use BM25 as your floor, not your ceiling. Build it first - it takes an afternoon with Lucene or a Python library - and measure everything against it. If a fancier system cannot beat a tuned BM25 on your own data, it is not better, whatever it scores elsewhere. In production, most strong systems run a hybrid: BM25 for exact and rare terms, embeddings for paraphrase, their rankings combined, and only then an AI model reasoning over the shortlist. The old formula's job is not to be clever. It is to make sure the right document is in the room.

Ground Truth. (2026, July 31). BM25 and lexical search: the keyword formula that keeps beating neural retrieval. Ground Truth. https://groundtruth.day/learn/bm25-and-lexical-search.html

@misc{groundtruth:bm25-and-lexical-search,
  title  = {BM25 and lexical search: the keyword formula that keeps beating neural retrieval},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/bm25-and-lexical-search.html}
}

Topics: retrieval · rag · search · bm25 · fundamentals