Retrieval That Can Show Its Work
Retrieve evidence first, synthesize second, and report uncertainty instead of hiding it.
By the end
You can combine keyword and semantic retrieval, assemble bounded context, generate cited answers, and expose confidence, freshness, and gaps.
Translate the question into a retrieval plan
Different questions require different evidence. What did we decide favors current decision pages and project records. What changed requires time-bounded sources and supersession history. Before search, identify entities, time window, user and project scope, requested output, and consequence. Saving that retrieval plan as a receipt prevents generic similarity search from pretending it understood the task.
Use each retrieval method for its real job
Keyword search is strong for names, IDs, and quotations. Semantic search finds conceptually related material with different vocabulary. Graph expansion follows explicit links from matched entities. Metadata filters enforce access, time, type, and status. Start with filters, combine lexical and semantic candidates, then expand only a few trustworthy neighbors. More context can make an answer worse by diluting relevant evidence.
Retrieve, rerank, then synthesize
Retrieve more candidates than you will show the model, rerank them for relevance, authority, freshness, and diversity, deduplicate mirror pages, and select a small evidence packet. Keep source IDs and location spans attached. Only then ask the model to answer using the packet; outside knowledge must be prohibited or clearly labeled.
- Apply access and project scope before retrieval.
- Budget context by evidence value, not file order.
- Keep conflicting evidence when conflict matters.
- Require citations adjacent to the claims they support.
Make uncertainty part of the result
A trustworthy result includes the retrieval path, sources used, source dates, and missing evidence. Confidence should come from observable conditions rather than the model's tone. Useful gap types include no results, narrow context, low similarity, stale sources, missing dates, conflicting claims, fallback retrieval, and uncited synthesis. A low-confidence answer can still help when it gives a concrete verification step.
Make fallback behavior visible
Semantic search, an embedding provider, or a synthesis model can fail while local files remain available. A strong system falls back to keyword search or direct navigation and reports the path used. Degraded output should lower confidence and explain the limitation. Keeping the fallback explicit prevents a partial result from masquerading as full-corpus semantic recall.
Working reference
Commands and patterns
[Starter repo after this lab] Request machine-readable evidence
npm run wiki:query -- --question 'What did we decide about storage?' --json
Runs the Chapter 5 query handler; the receipt includes sources, retrieval path, freshness, gaps, and the answer.
[Starter repo] Verify exact terms locally
rg -n -i 'storage|source of truth' vault/Wiki vault/Sources
Lexical search remains a strong verification and fallback tool for exact language.
When the happy path breaks
Failure modes
Symptom
Citations are topically relevant but do not support the exact claim.
Likely cause
Sources were attached after synthesis instead of during claim construction.
Recovery
Preserve spans through retrieval, require citations while drafting, and score citation support separately from topical relevance.
Symptom
A semantic-search outage produces an empty or confidently wrong answer.
Likely cause
There is no explicit fallback or degradation signal.
Recovery
Add lexical fallback, return the path used, lower confidence, and propose the next verification step.
Symptom
Old decisions outrank current ones because they sound similar.
Likely cause
Retrieval ignored status, time, and supersession.
Recovery
Filter and rerank by validity metadata, follow supersedes links, and surface conflicting history.
Hands-on lab
Build an answer-plus-proof function
Create a query path that remains honest when search, the model, or the corpus is incomplete.
- 01Write five answerable questions and two deliberately unanswerable questions.
- 02Implement keyword retrieval over title, body, tags, and source IDs.
- 03Optionally add embeddings while keeping keyword fallback.
- 04Return a stable evidence packet before adding synthesis.
- 05Generate an answer using only that packet and require inline citations.
- 06Add confidence, freshness, retrieval path, and gaps, then save all seven results.
Deliverable: A query function plus seven saved results demonstrating grounded answers, honest unknowns, and fallback behavior.
Reusable artifact
Answer contract
One stable shape for a CLI, dashboard, agent, or API.
question:
answer:
citations: [{ id, title, sourceId, location, similarity }]
retrievalPath: lexical | semantic | hybrid | graph
confidence: high | medium | low
freshness: { checkedAt, newestSourceAt, staleSourceIds }
gaps: [{ type, severity, message, citationIds }]
nextVerificationStep:Before you move on
Receipt checklist
- Fixed seven-question evaluation set.
- Raw candidates and final evidence packets.
- Answers with location-aware citations.
- Saved semantic-disabled and unanswerable results.
Primary sources
