LEANN, the MLSys 2026 best paper, skips storing embeddings and recomputes them at search time

Most vector databases store one embedding per chunk, and that is where the disk goes. LEANN stores almost none of them. It keeps a slimmed-down graph of which chunks are near which, and recomputes the few embeddings a search actually touches, on the fly. The paper behind it, from a Berkeley-led team of 13 authors including Ion Stoica, Matei Zaharia and Joseph Gonzalez, won the Best Research Paper award at MLSys 2026 in May. The open-source project (MIT licence, about 13,400 GitHub stars) shipped v0.3.8 on 28 September and is still being updated.

If you have read our vector database explainer and the one on index internals, this is the step that series did not cover: what if you simply do not keep the vectors?

How it works

• Graph only. LEANN builds a normal HNSW-style proximity graph (it builds on FAISS, with an optional DiskANN backend), then throws the embeddings away.
• Recompute at query time. During search it walks the graph and runs the embedding model only on the candidates it visits. A two-level search and dynamic batching keep that number small and the GPU busy.
• Prune the graph too. Its “high-degree preserving” pruning keeps the edges of well-connected hub nodes and cuts the rest, and the graph is stored in a compact CSR format.

The paper reports indexes below 5% of the raw data size, and up to 50 times smaller than a standard index. The README claims “97% less storage”, for example 60 million text chunks in 6 GB instead of 201 GB. Those are the project’s own measurements.

What you pay for it

Recomputing embeddings is not free. In the paper’s own table (RTX 4090, target 90% Recall@3), a retrieval takes 2.48 s on Natural Questions, 2.96 s on TriviaQA, 1.12 s on GPQA and 7.12 s on HotpotQA, against about 0.05 s for a plain in-memory HNSW index. The authors frame this against the LLM’s answer time: retrieval adds 10.6%, 14.7%, 1.6% and 23.4% respectively to end-to-end latency. The text says “less than 20%”; HotpotQA in the same table is above that, so judge by your own workload.

So LEANN is a laptop and workstation tool. It suits a private index of your mail, notes or a large codebase where disk matters and a second or two per query does not. It does not suit a server answering thousands of queries a second. The paper also covers Apple-silicon results and index updates.

Try it

uv pip install leann
leann build my-index --docs ./docs
leann search my-index "how do we rotate keys"

The repo also ships an MCP server (leann_mcp) for Claude Code. Its authors tested it against BM25 on 30 SWE-Bench Pro tasks with the same model and an 8,192-token retrieval budget: recall of the right code lines at the first retrieval was 24.2% against 11.4%, coverage 38.4% against 25.8%, with 8.4% fewer tokens. That is their test, and they note that better context “does not guarantee issue resolution”.

Rust experiments

There is no official Rust port. A few small independent experiments exist: leann-rs, islands (multi-repo code search over MCP) and vyakti, all MIT and all with a handful of stars. Treat them as experiments, not drop-in replacements.