Operate, Evaluate, and Recover
Make the wiki dependable with evaluations, quality metrics, backups, migrations, incident procedures, and visible trust.
By the end
You can detect degradation, recover from bad automation, and decide with evidence whether the knowledge system is improving.
Build evaluations from real work
Collect fixed questions across factual lookup, current decisions, relationship traversal, summarization, contradiction, and deliberate unknowns. Write expected sources and prohibited sources before tuning retrieval. Run the set after changes to schemas, prompts, chunking, embeddings, or graph logic, and keep difficult failures so the system remembers its mistakes.
Measure the chain, not just final prose
A fluent answer can hide weak retrieval. Measure whether the correct sources appeared, whether citations support claims, whether current records outrank superseded ones, whether unknowns are admitted, and whether scopes remain isolated. Also track ingest failures, orphan and broken-link counts, stale backlog, fallback rate, approvals, rollback frequency, latency, and cost.
Prove the vault can survive the application
Back up canonical files and originals independently from the search index. Restore them into a clean directory and rebuild derived indexes from documented commands. A backup is not proven until restore succeeds. Include encryption, key recovery, retention, and deletion propagation instead of testing only file copying.
Migrate knowledge like production data
Schema changes can alter hundreds of readable files. Implement version-to-version transformations with dry runs, counts, validation, samples, and rollback snapshots. Never ask an unconstrained agent to clean up the whole vault. After migration, verify links, rerun evaluations, and preserve a manifest showing every changed file.
Give the operator one trust surface
Show active retrieval path, last successful ingest, index freshness, unresolved lint issues, recent agent edits, pending approvals, and restore readiness. Define incidents such as source corruption, cross-user leakage, duplicate storms, broken citations, and silent fallback. Each runbook must stop further damage, preserve evidence, restore known-good state, and add a regression test.
Working reference
Commands and patterns
[Starter repo] Snapshot before a migration
git add vault/Sources vault/Wiki vault/System && git commit -m 'chore: snapshot wiki before migration'
A named snapshot provides an inspectable rollback point.
[Starter repo after Chapter 8 lab] Run the release gate
npm run wiki:lint && npm run wiki:eval && npm run wiki:restore-test
Runs the implemented lint, evaluation, and clean-restore handlers as one release gate.
When the happy path breaks
Failure modes
Symptom
Routine questions get worse after each ingest batch.
Likely cause
There is no fixed evaluation set or version comparison.
Recovery
Create golden questions with expected evidence and block material retrieval or citation regressions.
Symptom
Files are backed up but the team cannot restore a usable system.
Likely cause
The backup omitted sources, keys, schemas, or rebuild instructions.
Recovery
Run a clean-room restore that ends with a passing cited query.
Symptom
A migration edits hundreds of files before anyone notices an error.
Likely cause
There was no dry run, batch checkpoint, or stop condition.
Recovery
Generate a manifest, test a copy, apply in batches, validate each batch, and retain rollback.
Hands-on lab
Run a wiki release drill
Treat one schema or retrieval change like a production release.
- 01Create ten golden questions: six answerable, two conflicting or stale, and two unknown.
- 02Record expected and prohibited sources for each.
- 03Snapshot the vault and run baseline lint, evaluation, and isolation checks.
- 04Apply one bounded change through dry run and reviewed patch.
- 05Rerun the gate and compare every metric and question.
- 06Restore the baseline into a clean directory and prove one ingest and cited query.
- 07Write a release or rollback decision using the receipts.
Deliverable: A release packet with baseline, change manifest, evaluation comparison, restore proof, and ship-or-rollback decision.
Reusable artifact
LLM Wiki operating scorecard
A compact contract for quality, trust, and recoverability.
Corpus: sources, pages, latest ingest, invalid imports Integrity: checksum failures, missing sources, unresolved citations Structure: orphans, broken links, duplicates, stale pages Retrieval: recall, fallback rate, no-result rate, latency Answers: citation support, freshness, unknown calibration, conflicts Safety: denied isolation tests, approvals, sensitive access Operations: last backup, restore drill, schema version, rollback Change: recent patches, cost, decisions, open incidents
Before you move on
Receipt checklist
- Versioned golden-question suite.
- Baseline and post-change scorecards.
- Reviewed migration manifest and diff.
- Clean-directory restore ending in a cited answer.
- Written release or rollback decision.
Primary sources
