Capture Sources Without Losing Proof
Preserve originals, context, permissions, and the line between quotation and inference.
By the end
You can ingest a document safely by assigning identity, preserving the original, recording provenance, and controlling what may be derived.
Wrap every source in an evidence envelope
A file alone is not enough. Record where it came from, when it was captured, who owns it, who may use it, whether it is complete, and how integrity can be checked. Store that evidence envelope beside the original and carry its stable source ID into every derived page and citation.
- Origin URL, application, person, or system.
- Capture time and relevant event time.
- Filename, media type, byte size, and checksum.
- Access class, project scope, owner, and retention rule.
- Extraction method and known omissions.
Preserve originals; version interpretations
Never silently improve a source. OCR cleanup, speaker labeling, formatting repair, and summarization are transformations. Keep the original bytes and save each transformation as a derived artifact with its method and timestamp. A challenged claim should be traceable from wiki page to extraction span to original.
Chunk without destroying context
Long sources may need splitting before extraction or indexing. Fixed-size chunks can separate a qualification from its claim or a table row from its header. Prefer headings, paragraphs, transcript turns, pages, and table regions. Every chunk retains source ID, location range, section path, and neighboring context. Chunks accelerate retrieval; they never replace the original.
Permission is part of provenance
A source may be technically accessible and still be inappropriate to ingest. Personal messages, customer records, private browser pages, and licensed material require purpose and access boundaries. Label sensitivity at capture and filter by user, project, and class before content reaches a model. Redaction is a transformation and needs its own receipt.
Put a queue between capture and truth
New material should not instantly become authoritative. Use states such as captured, extracted, reviewed, promoted, rejected, and quarantined. Automate low-risk work such as hashing and file-type detection. Let models propose entities and claims, but preserve the distinction between proposed and accepted knowledge.
Working reference
Commands and patterns
[Starter repo after adding your source] Compute a checksum
shasum -a 256 vault/Sources/source-001/original.pdf
The digest lets you later prove whether the original bytes changed.
[Starter repo after adding your source] Inspect basic metadata
stat vault/Sources/source-001/original.pdf
Record size and capture details in the source envelope rather than relying on terminal history.
When the happy path breaks
Failure modes
Symptom
A cited URL changes or disappears.
Likely cause
The system stored only a location, not an authorized snapshot or capture record.
Recovery
Preserve a permitted original, capture time, origin URL, and stable source ID.
Symptom
A quotation cannot be found in the original.
Likely cause
A paraphrase was presented as verbatim source text.
Recovery
Separate quotations from summaries and verify exact strings and location spans.
Symptom
Private information appears in an unrelated answer.
Likely cause
Access scope was not enforced before retrieval.
Recovery
Attach scope at capture, filter before model calls, and test isolation with adversarial cross-project queries.
Hands-on lab
Create a verifiable source packet
Make one PDF, export, transcript, or image safe to derive knowledge from.
- 01Assign a stable source ID and create a source folder.
- 02Copy the original unchanged and calculate its SHA-256 checksum.
- 03Record origin, capture time, owner, scope, format, size, checksum, and retention.
- 04Create one derived extraction labeled with the method used.
- 05Split the extraction at meaningful boundaries with location pointers.
- 06Promote one supported claim and trace it back to the original.
Deliverable: A source packet with original, envelope, extraction, chunk map, and one source-backed claim.
Reusable artifact
Source envelope
A provenance record that travels with every captured item.
sourceId: title: originType: origin: capturedAt: eventAt: owner: accessClass: public | private | restricted projectScope: originalPath: mediaType: byteSize: sha256: extractionMethod: knownOmissions: retentionRule: reviewStatus:
Before you move on
Receipt checklist
- Checksum matching the preserved original.
- Source envelope with access and ownership fields.
- Derived extraction clearly labeled as derived.
- One claim traced from wiki page to original location.
Primary sources
