retrieval
Approximate nearest neighbor search: how a vector database finds a needle in a billion haystacks Lesson
Approximate nearest neighbor search is the algorithm that makes vector search fast enough to be useful - it finds the closest matches to a query embedding without comparing it against every item in the database, trading a small, tunable amount of accuracy for speedups of a hundred times or more. Every vector database and every retrieval-augmented system runs on it, and the accuracy it gives up is the hidden knob behind a lot of 'the retrieval just missed it' bugs.
BM25 and lexical search: the keyword formula that keeps beating neural retrieval Lesson
BM25 is a decades-old formula that ranks documents by how often a query's rare words appear in them, adjusted for document length - and it remains the baseline that modern AI retrieval systems have to beat, often unsuccessfully.
AskChem indexes 2.4 million individual chemistry claims instead of 147,000 papers News
A team at NYU built a live chemistry search system that breaks each paper into atomic claims, each carrying its own source identifier and verbatim quote, so a researcher or an AI agent retrieves the individual assertion rather than a ranked list of documents.
A decades-old keyword ranker beat the search agent once the document pile passed 10 million tokens News
In a controlled study that grew the same corpus across 28 nested sizes, the agent that browsed files won at small scale but spent 39 times more query tokens, and BM25 - a 1990s keyword ranking formula - overtook it around 10 million tokens and led by nearly 20 points at full scale.
AI search agents get better when relevance tells them where to look, not what to read News
Researchers at Tencent rebuilt relevance as a guide for how a search agent traverses a corpus rather than as a ranked list of documents, cutting the agent's tool calls by roughly a sixth while raising accuracy.
Two papers attack the same waste: coding agents rediscovering the same repository every session News
CodeNib builds reusable lexical, semantic and structural views of a repository per commit and cuts an agent's exploration tokens by 50 to 87%, while a companion benchmark finally measures the file-finding stage that patch-success scores hide.
Microsoft's new memory system lets AI agents remember more by storing less News
Memora keeps the rich detail of a conversation but searches it using tiny six-word labels, cutting the cost of remembering by up to 98 percent. The code is public.
Retrieval-Augmented Generation: giving a model an open book Lesson
A language model only knows what it learned in training, and it cannot cite sources. RAG fixes both by letting the model look things up in a real document store at answer time, then write its answer from what it found.
gget Tool
An open-source command-line and Python tool for querying genomic databases with exact, deterministic lookups. New benchmark work showed wrapping an AI agent around gget's 'virus' module lifted viral-sequence retrieval accuracy from as low as 17% to above 90% -- a concrete template for pairing models with hard tools.
T-Search Tool
An open agentic retriever you can try in the browser - it runs multi-round evidence gathering for questions that need several searches chained together rather than one lookup.
PyLate Tool
A training and retrieval library for late-interaction models, built on Sentence Transformers, for people who want to fine-tune a retriever on their own corpus rather than use an off-the-shelf embedding API.
NVIDIA Nemotron 3 Embed 8B Tool
8-billion-parameter retrieval encoder that turns queries and documents into normalized dense vectors for semantic search. NVIDIA claims state-of-the-art results on the multilingual RTEB leaderboard as of July 16; released under OpenMDW 1.1.
Microsoft Memora Tool
Open-source memory system for AI agents that stores rich content but searches it via tiny abstraction labels and cue anchors, cutting token cost on long-horizon tasks. Includes a distillable retriever.
LateOn Tool
LightOn's late-interaction counterpart to DenseOn - it keeps a vector per token instead of one per document, which costs more storage but retrieves noticeably better on hard queries.
FastPLAID Tool
A Rust engine for multi-vector search, the indexing layer that makes late-interaction retrieval fast enough to serve in production instead of only benchmarking well.
DenseOn Tool
A fully open 149-million-parameter dense retrieval model from LightOn for multilingual, long-context and code search, released with its training data and training code rather than weights alone.
AskChem Tool
A live chemistry search system that retrieves individual claims rather than papers - 2.4 million typed claims from 147,000 papers, each carrying a source identifier and a verbatim quote. Free web interface plus REST, SDK and MCP access so agents can query the same claim store.