Build the Ingestion Pipeline
Turn source material into proposed linked pages through a pipeline you can test one stage at a time.
By the end
You can design an ingest contract with extraction, validation, identity resolution, linking, dry runs, persistence, and rollback.
Build a pipeline, not one giant prompt
A reliable ingest flow is a chain of contracts: validate the source, extract text, propose knowledge, normalize fields, resolve identities, propose links, preview changes, persist approved files, update indexes, and write a receipt. Small stages make failures diagnosable. If the wrong entity appears, you can inspect extraction and identity resolution separately instead of rerunning an opaque agent.
Make the model return proposals, not prose
Ask for structured candidate entries matching the schema, then validate known categories, required source IDs, evidence spans, bounded summaries, and link proposals before any write. The model must be allowed to return no entry or unknown. Forcing an entry from every paragraph manufactures thin pages and false relationships.
- Candidate title, page type, and concise current summary.
- Evidence spans with source locations.
- Proposed aliases and relationships.
- Confidence and unresolved questions.
Resolve identity before creating pages
Normalize case and punctuation, search canonical titles and aliases, compare identifying attributes, and update an existing page when confidence is high. Never auto-merge ambiguous people or organizations. Produce a merge proposal with evidence for and against equivalence; an identity decision affects every future link and answer.
Preview the patch before persistence
A dry run should list pages to create, pages to update, links to add, conflicts, warnings, and untouched files without mutating the vault. An applied run should write atomically where possible, preserve the prior version, refresh indexes, and record source-to-page mappings. A successful model call is not a successful ingest; verification closes the operation.
Connect the pipeline to the starter CLI
The executable starter already separates proposal review from canonical persistence. With --dry-run, --baseline, and --stage together, src/ingest.ts copies the current vault into two disposable fixture snapshots and writes the candidate only to the proposed snapshot. The resulting directory diff is the review surface; vault/Wiki remains untouched. During this lab, refactor the handler into two explicit seams: planIngest(source, extractor) returns a validated IngestManifest, while applyIngest(manifest) persists exactly that reviewed plan. Begin with the deterministic extractor, then add an optional LLM Extractor adapter without changing fixtures, schemas, receipts, or write policy.
Working reference
Commands and patterns
[Starter repo] Build a reviewable ingest proposal
npm run wiki:ingest -- --source vault/Sources/source-001 --dry-run --baseline fixtures/baseline-vault --stage fixtures/proposed-vault --json
Rebuilds baseline-vault from the current canonical vault, clones it into proposed-vault, writes the candidate page only into proposed-vault, and emits a receipt whose staging.canonicalVaultChanged field is false. It never writes vault/Wiki.
[Starter repo] Review staged output
git diff --no-index fixtures/baseline-vault fixtures/proposed-vault
Compares the two snapshots populated by the previous command. On the untouched starter it shows Wiki/starter-source.md as a real proposed addition. Exit status 1 means differences were found and is expected during review.
When the happy path breaks
Failure modes
Symptom
Every source creates dozens of repetitive pages.
Likely cause
Extraction rewards quantity and has no usefulness or deduplication gate.
Recovery
Require a future-use reason, resolve existing pages first, and allow no durable output.
Symptom
A long source is only partially represented without warning.
Likely cause
Input was truncated but coverage was not reported.
Recovery
Chunk intentionally, record covered ranges and omissions, and never label partial ingestion complete.
Symptom
A failed run leaves half-created pages and broken links.
Likely cause
Writes occurred incrementally without a change manifest or recovery journal.
Recovery
Stage and validate the full patch, apply atomically or journal each write, and test interruption recovery.
Hands-on lab
Implement an idempotent text ingester
Build the smallest ingest flow that can be rerun safely on the same source.
- 01Define types for source input, candidate entry, proposed change, and ingest receipt.
- 02Extract one source into at most three candidates with evidence spans.
- 03Validate categories, fields, source IDs, and summary length.
- 04Resolve titles against existing files and aliases.
- 05Produce baseline and proposed fixture snapshots plus a dry-run manifest before writing.
- 06Apply it, rerun the source, and prove the second run updates or no-ops instead of duplicating.
Deliverable: A repeatable ingestion script, fixture source, generated pages, and first-run versus second-run manifests.
Reusable artifact
Ingest contract
A provider-neutral interface for testing each stage.
SourceRecord = { sourceId, sourcePath, mediaType, accessScope, sha256, extractionVersion }
Extractor = (source: SourceRecord) => Promise<CandidateEntry[]>
planIngest(source, extractor): Promise<IngestManifest>
applyIngest(manifest): Promise<IngestReceipt>
stageProposal(manifest, baselineRoot, proposalRoot): rebuild both fixture snapshots and write only the proposal root
1. Validate source envelope and integrity
2. Extract text with location spans
3. Propose entries and evidence
4. Normalize titles, categories, dates, and tags
5. Resolve canonical pages and aliases
6. Propose links and identify conflicts
7. Emit a dry-run change manifest and a reviewable baseline-versus-proposal diff
8. Apply approved writes and refresh indexes
Output: created, updated, unchanged, rejected, warnings, receiptIdBefore you move on
Receipt checklist
- Fixture source and expected extraction.
- Dry-run manifest listing every proposed file change.
- First-run and second-run results proving idempotency.
- Invalid fixture rejected without modifying the vault.
Primary sources
