Contents
- 01From Chatbot to Answer Engine
- 02Model the Personal Knowledge Graph
- 03Engineer Trust, Provenance, and Uncertainty
- 04Build a Self-Wiring Ingestion Flow
- 05Retrieve, Traverse, and Synthesize
- 06Build the Dream and Consolidation Cycle
- 07Secure and Operate a Multi-User Brain
- 08Evaluate Trust and Ship the Product
Evaluate Trust and Ship the Product
Measure retrieval, graph value, citation support, gap honesty, freshness, security, and operator usability before calling the brain reliable.
By the end
You can run a complete evaluation suite, set release gates, design a trust-centered interface, and make an evidence-backed shipping decision.
Build a matrix, not a single accuracy score
Test direct fact lookup, multi-hop relationships, current versus historical truth, conflicting evidence, stale sources, ambiguous identity, no-result questions, permission denial, and degraded services. One average score can hide a catastrophic privacy or citation failure. Report each capability and risk separately.
Measure retrieval and graph contribution
Track whether expected evidence appears in the top candidates, whether irrelevant sources are excluded, and whether graph expansion adds necessary evidence. Compare lexical-only, vector-only, hybrid, and hybrid-plus-graph runs. If graph traversal adds cost and noise without improving answerable questions, simplify it.
Score citation support and gap calibration
Evaluate each material claim for citation presence and support. Measure whether stale and contradictory evidence is surfaced and whether deliberately unknown questions are refused appropriately. Gap quality matters: a useful gap names what is missing and the next verification step, not merely low confidence.
Design for inspection and correction
A user should be able to expand citations, see source dates, inspect graph paths, correct identity, dispute a claim, approve consolidation, and export or delete data. The default answer stays readable while proof remains one step away. Operator views should prioritize pending decisions and failures rather than decorative graph animation.
Ship through gates and monitor drift
Set explicit thresholds for source isolation, citation support, unknown calibration, fallback behavior, restore, latency, and cost. Version corpus fixtures, prompts, models, and schema. After release, sample real traces, track regressions by version, review consolidation changes, and convert every incident into a new fixture and test.
Working reference
Commands and patterns
Export and replay local evaluation captures
export GBRAIN_CONTRIBUTOR_MODE=1 gbrain eval export --since 7d > baseline.ndjson gbrain eval replay --against baseline.ndjson --limit 50
Contributor capture is off by default. When explicitly enabled, real-user captures remain local; replay a local export to detect behavior changes without publishing private data.
Run the registered hand-labeled correctness gate
gbrain eval gate --qrels qrels.json
Create qrels.json from the expected evidence in your evaluation matrix, then run the registered gate subcommand. The pinned repository contains bench-publish.ts, but gbrain bench is not registered in its CLI, so this guide does not claim that unreachable command works.
Record operational readiness
gbrain doctor --json gbrain stats gbrain embed --stale --dry-run gbrain check-update --json
Captures health, corpus shape, the count of stale embeddings without re-embedding, and update status as separate operational evidence before release.
When the happy path breaks
Failure modes
Symptom
The product reports high answer accuracy but fails unknown and conflict cases.
Likely cause
The test set contains only answerable lookup questions.
Recovery
Add unknown, ambiguous, stale, contradictory, unauthorized, and degraded-service scenarios.
Symptom
The graph looks impressive but does not improve results.
Likely cause
No ablation compares answers with and without graph expansion.
Recovery
Run configuration comparisons and keep graph paths only where they improve evidence coverage or explanation.
Symptom
A user cannot correct a wrong entity or claim.
Likely cause
The product optimized for answer generation rather than governed knowledge.
Recovery
Add inspect, dispute, correct, approve, export, and delete workflows with receipts.
Hands-on lab
Run a release-candidate evaluation
Evaluate one complete engine across trust, quality, security, usability, cost, and recovery.
- 01Create at least twenty questions spanning all evaluation categories.
- 02Record expected evidence, allowed gaps, prohibited sources, and acceptance thresholds, then encode retrieval relevance judgments in qrels.json.
- 03Run lexical, hybrid, and hybrid-plus-graph configurations.
- 04Score retrieval, claim support, freshness, conflicts, unknowns, latency, and cost.
- 05Run cross-user, provider-outage, and restore tests.
- 06Have one new user inspect and correct a deliberately wrong claim.
- 07Write a release, limited-pilot, or rollback decision from the receipts.
Deliverable: A release report with configuration comparison, failure analysis, user correction proof, and explicit decision.
Reusable artifact
GBrain release scorecard
The evidence required before a knowledge engine earns broader trust.
Corpus and fixture version Direct lookup and multi-hop retrieval Graph contribution versus no-graph baseline Claim citation presence and evidence support Freshness and supersession accuracy Conflict surfacing and unknown calibration Cross-user isolation and sensitive-provider policy Fallback, latency, cost, and availability Backup, restore, index rebuild, and rollback Human inspection, correction, approval, export, and deletion Decision: release | limited pilot | rollback, with reasons
Before you move on
Receipt checklist
- Versioned twenty-question evaluation matrix.
- Configuration ablation including no-graph baseline.
- Claim-level citation support results.
- Unknown, conflict, privacy, and degraded-service results.
- Human correction workflow receipt.
- Evidence-backed release decision.
Primary sources
