rag
Approximate nearest neighbor search: how a vector database finds a needle in a billion haystacks Lesson
Approximate nearest neighbor search is the algorithm that makes vector search fast enough to be useful - it finds the closest matches to a query embedding without comparing it against every item in the database, trading a small, tunable amount of accuracy for speedups of a hundred times or more. Every vector database and every retrieval-augmented system runs on it, and the accuracy it gives up is the hidden knob behind a lot of 'the retrieval just missed it' bugs.
One planted document flipped more than half of deep-research reports to a false conclusion News
Researchers built 5,933 credible-looking but factually false documents and slipped exactly one into the retrieval pool of several deep-research agents; the rate at which final reports endorsed the false conclusion went from zero to 54.7%.
BM25 and lexical search: the keyword formula that keeps beating neural retrieval Lesson
BM25 is a decades-old formula that ranks documents by how often a query's rare words appear in them, adjusted for document length - and it remains the baseline that modern AI retrieval systems have to beat, often unsuccessfully.
A decades-old keyword ranker beat the search agent once the document pile passed 10 million tokens News
In a controlled study that grew the same corpus across 28 nested sizes, the agent that browsed files won at small scale but spent 39 times more query tokens, and BM25 - a 1990s keyword ranking formula - overtook it around 10 million tokens and led by nearly 20 points at full scale.
AI search agents get better when relevance tells them where to look, not what to read News
Researchers at Tencent rebuilt relevance as a guide for how a search agent traverses a corpus rather than as a ranked list of documents, cutting the agent's tool calls by roughly a sixth while raising accuracy.
Retrieval-Augmented Generation: giving a model an open book Lesson
A language model only knows what it learned in training, and it cannot cite sources. RAG fixes both by letting the model look things up in a real document store at answer time, then write its answer from what it found.
Agent memory: how an AI remembers you after the conversation ends Lesson
Why most AI assistants have amnesia, the difference between short-term context and real long-term memory, and why remembering you is both what makes agents useful and what makes them a privacy risk.
cognee Tool
An open-source persistent-memory layer for AI agents with remember, recall, forget, and improve operations over a graph-plus-vector store, able to run graph relations, embeddings, session cache, and metadata in a single Postgres instead of four services.
RAGFlow Tool
An open engine for building AI question-answering over your own files and documents.
PyLate Tool
A training and retrieval library for late-interaction models, built on Sentence Transformers, for people who want to fine-tune a retriever on their own corpus rather than use an off-the-shelf embedding API.
MinerU Tool
Open-source tool that converts complex PDFs and office files into clean markdown and structured data that AI models can read reliably. Run it yourself for free, with nothing leaving your machine.
LateOn Tool
LightOn's late-interaction counterpart to DenseOn - it keeps a vector per token instead of one per document, which costs more storage but retrieves noticeably better on hard queries.
Firecrawl Tool
A hosted API that crawls, scrapes and structures web pages into clean text for agents and retrieval pipelines, handling the JavaScript rendering and rate limiting you would otherwise build yourself.
DenseOn Tool
A fully open 149-million-parameter dense retrieval model from LightOn for multilingual, long-context and code search, released with its training data and training code rather than weights alone.