The retrieval contract
Given a query, corpus, access policy, and latency budget, return a small evidence set that contains the material needed to answer.
Recall first. Order second. Ground always.Interactive RAG evaluation tool
Find evidence before the model answers
Change hybrid retrieval, candidate depth, metadata filtering, top k, and reranking. The live benchmark computes evidence metrics from labeled teaching corpora while the 3D scene shows which documents survive each stage.
Recall@k0.000 of 0 found
MRR0.00first relevant rank
nDCG@k0.00graded ordering
Precision@k0.000 useful chunks
Routing
Pipeline stage
RAG is an evidence-routing system, not a truth switch
Retrieval-augmented generation separates two jobs. A retrieval system chooses a small evidence set from a larger collection. A generator then uses that selected context to produce an answer. The quality of the answer is bounded by both stages: a careful generator cannot cite a policy it never received, and perfect retrieval does not force a model to interpret evidence correctly.
This lab makes the first job inspectable. Each scenario contains ten short chunks, synthetic semantic scores, visible metadata, and human relevance grades from zero to three. BM25 lexical scores are computed in your browser from the actual query and chunk text. The semantic values stand in for a dense encoder. They are deliberately labeled as teaching scores rather than measurements from a named model.
The semantic-weight slider blends normalized BM25 with the teaching dense score. At zero percent, exact words dominate. At one hundred percent, only semantic similarity remains. The candidate-pool control cuts the first-stage ranking before reranking, demonstrating an important constraint: a reranker cannot rescue a relevant document that retrieval excluded.
Turn on Verified to apply metadata before ranking. This can improve precision by removing untrusted notes, but a bad filter can destroy recall. Turn reranking off to inspect the first-stage ordering. Turn it back on to apply a stronger joint query-document teaching score only to the candidate pool, then compare the live Recall@k, MRR, nDCG@k, and Precision@k.
The visual scene follows the same arrays shown in the inspector. Corpus nodes begin in a shared index. Retrieval fans beams according to first-stage score. Reranking moves the candidate set into a new order. Context sends only the final top k toward the generator. Nothing is hidden behind a decorative animation.
Labels stay out of scoring
Relevance grades are used only after ranking to calculate evaluation metrics. The benchmark never feeds the answer key into BM25, hybrid scoring, or reranking.
Candidate depth is a ceiling
If a relevant chunk ranks below the candidate cutoff, later reranking cannot see it. First-stage recall sets the maximum possible downstream quality.
Filters encode policy
Metadata filters can enforce tenant, permission, freshness, language, geography, or source rules before similarity scoring. Their false negatives matter.
Top k spends context
Larger k can recover more relevant evidence, but it also adds tokens, weak context, latency, and opportunities for contradictory passages.
Measure retrieval before judging the generated prose
Evaluate the retriever separately
End-to-end answer scores mix retrieval, prompting, model behavior, citation formatting, and evaluator noise. A retrieval benchmark isolates whether the necessary source material entered the context at all.
Use graded judgments where utility differs
A canonical policy may be indispensable, a supporting procedure may be helpful, and an anecdotal note may only be marginally relevant. Binary labels collapse those differences. nDCG preserves them.
Build queries from real failure modes
A benchmark should include paraphrases, abbreviations, stale records, near-duplicate chunks, permission boundaries, multi-hop questions, and lexical traps. Easy keyword matches produce misleading confidence.
Report slices, not only averages
Aggregate scores can hide failures for a tenant, language, document type, time range, or safety-sensitive workflow. Track slices that correspond to product risk and user experience.
Keep a no-answer path
If retrieval cannot find sufficient evidence, the system should abstain, ask a clarifying question, or escalate. Forcing an answer converts retrieval uncertainty into hallucinated certainty.
Open each stage of the retrieval pipeline
Index
Split source material into retrieval units, preserve source identity, attach permissions and timestamps, and compute representations. Chunk boundaries influence whether one result contains a complete claim or a fragment that depends on missing neighbors.
IndexRetrieve
Use lexical, dense, sparse, or hybrid search to maximize candidate recall under a latency budget. Approximate nearest-neighbor indexes trade a small amount of recall for large speed gains on high-dimensional vectors.
SearchFilter
Apply access, tenant, geography, source, and freshness policy. Filtering early reduces exposure and cost, but every rule needs tests for accidental exclusion of relevant evidence.
PolicyRerank
Score the query and each candidate jointly with a stronger model or feature set. Reranking usually improves precision and ordering, but its work scales with candidate-pool depth.
OrderAssemble
Deduplicate, diversify, fit the token budget, retain citations, and represent conflicts. Context assembly is an optimization problem, not simply concatenating the top k strings.
ContextThree benchmark cases expose different failure modes
"Can I return a damaged item after 30 days?"
This query needs an exception policy, not only the standard return window. Dense retrieval helps connect damaged goods with warranty language; lexical retrieval catches the explicit 30-day wording. A strong reranker should place the canonical damaged-item policy above a high-scoring complaint note.
A production RAG evaluation checklist
Define the task, corpus boundary, and no-answer behavior before selecting metrics.
Keep human relevance judgments separate from every ranking feature.
Measure first-stage recall at the exact candidate depth sent to reranking.
Evaluate final top-k ordering with graded labels and nDCG.
Test permission, tenant, freshness, and source filters for false exclusions.
Include lexical traps, paraphrases, abbreviations, and near duplicates.
Track latency and cost at retrieval, reranking, and context assembly.
Measure answer citation correctness separately from retrieval quality.
Inspect performance slices for languages, tenants, source types, and risk classes.
Version corpora, labels, chunking, embedding models, indexes, and prompts.
Frequently asked questions
Are the semantic and reranker scores from a real model?
No. They are deterministic teaching values designed to expose common ranking tradeoffs. BM25 is computed from the visible query and chunk text. Relevance labels are used only to calculate metrics. A production benchmark should run the exact embedding model, index, filters, and reranker used by the application.
Why not retrieve directly with the reranker?
Joint query-document scorers are usually more expensive than vector or lexical retrieval because they process each pair together. A fast first stage narrows millions of candidates to a manageable pool, and a stronger reranker spends computation only on that pool.
Does a higher similarity score mean a claim is true?
No. Similarity estimates relevance under a representation and objective. It does not guarantee authority, freshness, factual correctness, permission, or completeness. Source policy and downstream verification remain necessary.
Should top k be as large as the context window permits?
Usually not. More context can increase recall, but irrelevant or contradictory passages can reduce answer quality and increase latency and cost. Tune k against retrieval and end-to-end evaluations rather than treating context capacity as a target.
What is the difference between recall and precision?
Recall measures how much of all relevant evidence was recovered. Precision measures how concentrated the returned set is with relevant evidence. Increasing k often raises recall and lowers precision, which is why both should be observed.
Can metadata filtering happen after retrieval?
It can, but access-control filters should generally constrain candidates before protected content is returned or processed. Some vector systems support pre-filtering; others approximate it or combine filtered sub-indexes. Test both correctness and performance.
How do I evaluate a question with no relevant document?
Add explicit unanswerable examples and score whether the system abstains or asks for more information. Standard recall is undefined when no relevant item exists, so report a separate rejection or no-answer metric.
Primary research and implementation references
Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksLewis and colleagues describe combining parametric generation with retrieved non-parametric memory.
Dense Passage Retrieval for Open-Domain Question AnsweringKarpukhin and colleagues study dual-encoder dense retrieval and in-batch negative training.
Passage Re-ranking with BERTNogueira and Cho demonstrate joint query-passage reranking with a pretrained transformer.
BEIR: A Heterogeneous Benchmark for Zero-shot EvaluationBEIR evaluates retrieval systems across diverse datasets and domains.
Reciprocal Rank FusionCormack, Clarke, and Buettcher present a robust method for combining ranked retrieval results.
Introduction to Information RetrievalManning, Raghavan, and Schutze provide foundational treatment of indexing, ranking, and evaluation.
Three.js DocumentationThe WebGL library used for the spatial evidence-routing visualization.
Super personal AI agentsSee how practical agents connect evidence, tools, approvals, and user-facing workflows.
Evaluate the evidence path before trusting the answer.
Reliable retrieval is observable: the corpus is scoped, candidates are inspectable, filters are tested, rankings are measured, and insufficient evidence produces an explicit stop rather than invented certainty.
Explore evidence-aware computer use