Skip to content

Data Contracts

Ingest has five distinct contract layers: canonical source metadata, canonical corpus snapshots, functional source records, in-process chunk values, and serialized boundary models. They are related, but they are not interchangeable.

Canonical Source Metadata

normalize_source_metadata() resolves bibliographic metadata by declared source type, never by mapping merge order or caller iteration order. For bibliographic fields the fixed precedence is:

user override
  > corpus lock
  > acquisition receipt
  > embedded/parser metadata
  > filename fallback

Records at the same level are ordered by their provenance label and exact record hash, so changing input iteration order cannot change the result.

Discovery remains authoritative for the admitted relative path, media type, content SHA-256, and byte length. A filename stem is used as a title only when no higher source supplied a title; reviewed metadata is therefore never replaced by a filename guess.

The bijux.canon.ingest.source_metadata.v2 manifest retains every contributing record's source type, label, exact record hash, optional source-content hash, and contributed fields. Each raw field also retains its original and normalized value. selected_values identifies the winning contribution for each field, while conflicts records every normalized disagreement with that selection. Equivalent Unicode, URI, DOI, date, language, license, and author forms do not create false conflicts.

Identity evidence fails closed. A declared record identity must match its canonical record, and any supplied source format, hash, or byte length must match the admitted immutable source. A DOI and DOI-based canonical URI that identify different works are rejected with MetadataIntegrityError; ordinary metadata disagreement remains visible as a conflict rather than being overwritten.

Every supported parser contributes one embedded_parser record bound to its parser manifest, source hash, format, and byte length. JATS and HTML contribute their structured citation fields; PDF and DOCX contribute embedded properties and semantic titles; Markdown contributes bounded top-level front-matter fields; and plain text contributes an identified title block when present. These records use the same resolver as reviewed records, so an unlocked corpus selects real parser truth while a lock can override it without erasing the parser value or disagreement.

Corpus Lock Resolution

Canonical directory ingest checks for corpus.lock.json in the document root, its parent, and—when ingesting a conventional corpus/sources directory—the portfolio root. corpus_lock_path (Python and HTTP) or --corpus-lock (CLI) selects an explicit lock. When no lock is present, ingest remains available and the metadata manifest retains lower-provenance discovery and parser records, using the filename only when no parser or supplied record exposes a title.

The loader accepts the governed parser-source and research-corpus lock schemas. Before parsing, it verifies the canonical lock identity, aggregate and per-file counts and sizes, portable paths, exact source hashes, compatible media types, license expression and URL, and linked source/acquisition identities. The parser portfolio also verifies its source JSON, acquisition receipts, transport checksums, and license-evidence checksums. The research portfolio verifies its materialization manifest, exact manifest hash, portfolio identity, and linked acquisition identities. A stale, partial, extra, malformed, ambiguous, escaped, or tampered chain fails with CorpusLockError and a stable issue code. Preparation manifests use bijux.canon.ingest.corpus_preparation.v2 and retain the portable lock schema, identity, source count, and automatic-or-explicit discovery mode. Snapshot assembly continues to accept identity-valid v1 preparations as unlocked legacy inputs. Ingest result manifests expose the same portable lock summary, or {"status": "absent"}; neither persisted form leaks the host path of the selected lock.

Bounded Directory Discovery

bijux.canon.ingest.discovery.v2 records the include/exclude and symlink policy together with the exact maximum depth, enumerated entries, admitted files, bytes per file, total admitted bytes, and elapsed seconds. These limits also appear in bijux.canon.ingest.corpus_snapshot_configuration.v3, so a snapshot identity cannot silently reuse content discovered under a different resource policy.

Traversal opens directories and regular files without following a path that was replaced by a symlink after enumeration. It compares device/inode and size/time identity around reads, resolves permitted symlinks strictly within the canonical root, detects directory cycles, rejects special files, and orders portable relative names by UTF-8 bytes. Oversize files are refused from stat evidence before their content is opened. Exhausting any traversal budget is an incomplete typed discovery outcome, never a smaller successful corpus.

Canonical Corpus Snapshot

Every admitted snapshot document carries an ingest-owned bijux.canon.ingest.document_citation_lineage.v1 graph. It binds the immutable source SHA-256, document identity, parser name/version and parser-manifest identity to every normalized mapping and semantic chunk. Each source-to-document, document-to-mapping, and mapping-to-chunk edge has its own content identity.

A chunk can combine several semantic blocks. Its lineage therefore contains an ordered locator segment for every contributing mapping rather than inventing one locator for the combined text. Segment spans use Unicode code-point coordinates in the normalized chunk and retain the actual format locator: JATS element path, PDF page text span, HTML DOM path, Markdown or text line span, or OOXML package-part/block coordinates. Exact segment text remains in the sibling chunk record and is bound by SHA-256; repeated text is resolved by the structural locator, never by searching for the quotation.

Resolution verifies the whole source payload identity, parser manifest, document and mapping identities, normalized offsets, exact text hash, and the format locator. Missing and multiply resolving locators are typed locator_unavailable and locator_ambiguous refusals. PDF and OOXML locators identify text in a deterministic parser representation bound to the exact source payload; they do not pretend compressed container bytes are direct text coordinates.

Immutable Snapshot Publication

bijux.canon.ingest.corpus_publication.v2 is the active-generation manifest. It binds the canonical snapshot bytes and snapshot identity to a bijux.canon.ingest.corpus_object_relation.v1 identity, plus explicit source and derived relation-entry counts. The relation enumerates the exact source bytes, canonical snapshot, parsed-document and metadata manifests, citation-lineage graph, normalized mappings, and semantic chunks. Every entry records its kind, document identity, stable domain identity, byte length, and SHA-256 content address.

Object counts describe logical relation entries; identical byte payloads may share one physical content-addressed object. A reader accepts the publication only after canonical JSON, snapshot identity, relation identity, entry counts, every reachable object digest, and the snapshot-object binding all verify. Version 1 generation manifests remain readable for existing stores but do not claim the version 2 relation guarantees.

The relation also retains integrity-bound snapshot-reuse bundles. They carry the complete logical source identity and every typed parser, metadata, mapping, and chunk member needed to reconstruct unchanged documents after restart. Restoration must reproduce the exact snapshot bytes and identity; missing, ambiguous, or altered members refuse reuse. Existing generations without these bundles remain readable and replayable, but the first ingest after upgrade performs a full derivation into the reuse-capable configuration identity.

bijux.canon.ingest.result.v2 reports initial, unchanged, or changed in disposition. When an active prior generation was restorable, delta binds the previous and current snapshot identities and canonically records added and deleted documents, source modifications, renames, tombstones, chunk invalidations/additions, and rejection changes. An unchanged result has equal snapshot identities and empty change sets; it is not inferred from timestamps.

Source Record

RawDoc represents one source row:

Field Type Meaning
doc_id string caller-supplied document identity
title string source title
abstract string text normalized and segmented by the pipeline
categories string source classification text

CleanDoc carries the same fields after configured normalization. The pipeline does not allocate a new document identity during cleaning.

Retrieval Chunk

ChunkWithoutEmbedding is the segmentation contract. Chunk extends it with an immutable embedding tuple.

Field Rule
doc_id identifies the source record
text contains the exact segmented text
start, end integer source offsets; start >= 0 and end >= start
chunk_index non-negative ordinal within the source document
title, category optional source context
metadata read-only mapping for non-structural context
embedding_spec optional dimensionality and model contract
embedding final numeric vector, present on Chunk
chunk_id SHA-256 identity derived from source ID, offsets, and text

Metadata and the embedding specification do not participate in chunk_id. Changing either may still change retrieval behavior, so an index fingerprint must cover the full vector set and configuration rather than relying on chunk IDs alone.

Serialized Edge Model

The strict Pydantic ChunkModel is a versioned edge shape used by the serialization adapters. Its contract is intentionally narrower:

{
  "version": 1,
  "text": "a retrieval-ready passage",
  "metadata": {"source": "catalog.csv"},
  "embedding": [0.125, -0.25]
}

The model rejects unknown fields, coercion, empty text, non-finite vector values, vectors longer than 8192 entries, and values outside the accepted numeric range. embedding may be omitted; when present, it must be non-empty. The computed text length is available in memory but excluded from canonical serialization.

The core pipeline's retrieval Chunk and the edge ChunkModel serve different purposes. The serialization adapters operate on the functional-programming core chunk, not the retrieval chunk described above. to_core_chunk() retains text and metadata; from_core_chunk() returns those same fields. embedding is not transferred in either direction. Document identity, offsets, ordinal, title, category, and embedding specification are outside this adapter.

Identity and Projection Map

Every projection has a deliberate loss boundary:

Transition Preserved Recomputed or changed Not carried
RawDoc to CleanDoc document ID, title, categories normalized abstract original abstract bytes
CleanDoc to retrieval chunk document ID, selected text, source offsets ordinal and chunk_id unselected document text
unembedded to embedded chunk all chunk fields embedding tuple nothing when the same chunk is enriched
ChunkModel to FP core chunk text, metadata core path becomes empty embedding and retrieval identity
FP core chunk to ChunkModel text, metadata version defaults to 1 path and embedding

Only the first three transitions belong to the retrieval pipeline. The last two are an edge-model interoperability path. Do not use that path to checkpoint a retrieval chunk or to move vectors between systems.

File Boundaries

The document CLI reads CSV records and writes one successful chunk per JSONL line. Expected failures are returned or rendered separately; they are not interleaved with successful rows in the output file. Treat the output as a projection with an accompanying run context:

  • write it to a new path or replace it atomically at the surrounding workflow;
  • retain the input and effective configuration when reproducibility matters;
  • validate every line and the expected row count before building an index; and
  • do not infer an embedding model solely from vector length.

Expected boundary failures are represented with Result values. Unexpected programming failures remain exceptions so corrupted or partial outputs are not mistaken for successful domain results.

Compatibility Rules

A change is compatibility-sensitive when it alters field meaning, stable ID inputs, offset interpretation, serialization version, validation limits, or the distinction between omitted and empty embeddings. Add a new version or an explicit migration path for such changes; silently accepting both meanings would make replay evidence ambiguous.

See Execution Model for how these values move through the pipeline.