News · 2026-07-31

A decades-old keyword ranker beat the search agent once the document pile passed 10 million tokens

A controlled scaling study published July 29 found that BM25, a keyword ranking formula from the 1990s, overtakes an agent that browses a filesystem once a document collection passes roughly 10 million tokens - and then leads at every larger size, by a margin approaching 20 points at full scale. The agent won at the smallest sizes, but it spent 39 times more query tokens to do it. The paper is titled, without hedging, "BM25 Wins at Scale."

Key facts

• BM25 overtakes the File-System Agent at around 10 million corpus tokens and leads at every larger tier, with a gap approaching 20 points at full scale.
• The agent uses 39 times more query tokens than BM25 at the study's fixed baseline corpus.
• Posted to arXiv on July 29, 2026, by Pengyu Wang, Benfeng Xu and colleagues.
• Primary source: BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms.

The design is the reason this result carries weight. Most comparisons of retrieval methods test each approach on its own benchmark at one corpus size, which tells you almost nothing about what happens as your document pile grows. This study instead varies corpus size across 28 strictly nested tiers spanning roughly a 450-fold range - each larger tier literally contains the smaller one - while keeping the questions, the relevant documents, the deliberately misleading documents, the reader model and the judging protocol fixed. Only the size of the haystack changes.

Four approaches were compared: BM25 lexical retrieval, dense retrieval using embeddings, graph-based indexing, and agentic search via a File-System Agent that reads and navigates the corpus the way a person would.

The result is a crossover, not a winner. The authors are careful about this: "The results reveal a scale-dependent crossover rather than an unconditional winner." At the smallest shared tiers, the browsing agent leads. Then the curve bends.

The mechanism is intuitive once you see it. A browsing agent explores in sequence - open a directory, read a file, decide where to look next - and each of those steps costs generated tokens and wall-clock time. BM25 does the opposite: it builds an index once, then ranks the entire collection in a single pass and hands back a shortlist. It is the difference between walking a library aisle by aisle and reading the card catalogue. Walking works fine in a small room. In a warehouse, the catalogue wins by an increasing margin, because the catalogue's cost barely moves while the walking cost grows with the floor space.

Worth noting is what the crossover point implies for real deployments. Ten million tokens is not a large corpus by enterprise standards - it is roughly a mid-sized company's internal documentation, or a few years of one team's support tickets. Almost every production retrieval system anyone is building today already sits well past the point where, in this study, the browsing agent had stopped being the better choice and had started being both slower and less accurate. The tier where the agent wins is the tier where you could have fit everything in the context window anyway.

Graph-based retrieval fared worst on practical grounds: the paper reports it "encounters construction walls before deployment scale," meaning the cost of building the graph became prohibitive before the corpus reached realistic size. Dense retrieval stayed cheap but less accurate. BM25 anchored the low-cost end of the trade-off curve without needing any language model to build its index at all.

The authors' concluding line is the one worth carrying: "corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it." That is not an argument against agents. It is an argument about ordering - shortlist first, reason second.

The honest caveat is scope. This runs on one benchmark, EnterpriseRAG-Bench, built around a fictional company with a fixed question set and planted distractor documents. It does not test hybrid retrieval, reranking, late-interaction methods, or every agent architecture, and it uses one reader model throughout. "BM25 wins under these conditions once scale is large enough" is what the evidence supports; "BM25 always wins" is not. The paper's own earlier title, before revision, was the more modest Which RAG Paradigm Wins at Scale? - and that version of the question is still the better one to hold in mind.

Primary source, verified: read the paper → (arXiv 2607.26497)

Ground Truth. (2026, July 31). A decades-old keyword ranker beat the search agent once the document pile passed 10 million tokens. Ground Truth. https://groundtruth.day/news/bm25-overtakes-the-search-agent-at-ten-million-tokens.html

@misc{groundtruth:bm25-overtakes-the-search-agent-at-ten-million-tokens,
  title  = {A decades-old keyword ranker beat the search agent once the document pile passed 10 million tokens},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/bm25-overtakes-the-search-agent-at-ten-million-tokens.html}
}

Topics: rag · retrieval · agents · benchmarks · search · inference-cost