Skip to content

State and Persistence

Ingest transformations are predominantly value-to-value operations. Durability appears only where the caller selects a file, cache, index, or service boundary. This keeps retention visible, but it also means that a collection of individually valid files is not automatically one coherent corpus generation.

State Classes

flowchart TD
    source["caller-owned source"] --> values["RawDoc and CleanDoc values"]
    values --> chunks["Chunk values"]
    chunks --> jsonl["prepared JSONL"]
    chunks --> index["MessagePack reference index"]
    chunks --> cache["optional disk cache"]
    index --> http["process-local HTTP registry"]

    values -. "ephemeral" .-> chunks
    jsonl -. "durable file" .-> generation["published corpus generation"]
    index -. "durable file" .-> generation
    cache -. "reconstructible" .-> generation
    http -. "ephemeral" .-> generation
State Lifetime Identity Recovery posture
Python documents and chunks process or iterator lifetime document and chunk IDs, normalized offsets rerun from retained source and configuration
canonical corpus snapshot caller-selected publication root snapshot, relation, and content-object SHA-256 identities verify active manifest, generation, relation, and every reachable object; otherwise recover the previous valid generation
prepared JSONL caller-controlled file lifetime per-record identity plus generation metadata supplied by caller validate, or rebuild the complete generation
BM25 or NumPy-cosine index caller-controlled file lifetime schema version and index fingerprint load and validate; rebuild from prepared records on failure
disk cache entry configured cache retention namespace, caller version, SHA-256 key filename evict and recompute
HTTP index registry application-process lifetime index_id derived from index fingerprint rebuild or reload after restart

The library does not select a global data directory. Paths, permissions, backup, cleanup, and retention are deployment decisions.

File Commit Boundaries

Prepared JSONL and cache entries publish through a temporary sibling and atomic replacement on the same filesystem. This prevents ordinary readers from seeing a partially written target. It does not make a transaction across prepared records, index files, configuration, observations, and downstream registrations.

flowchart LR
    source["source snapshot"] --> stage["fresh generation directory"]
    config["transform and model config"] --> stage
    stage --> chunks["chunks.jsonl"]
    stage --> index["reference index"]
    stage --> manifest["generation manifest"]
    chunks --> verify{"all artifacts validate?"}
    index --> verify
    manifest --> verify
    verify -->|no| quarantine["retain or remove staged generation"]
    verify -->|yes| publish["atomically publish generation pointer"]

For a multi-artifact corpus, write into a fresh generation directory. Record source and normalized digests, transformation configuration, embedder identity, schema versions, index fingerprint, counts, and terminal observations. Validate every required artifact before changing the application-visible generation pointer.

corpus build --publish-root implements that boundary for canonical ingest. It stores the exact source bytes, canonical snapshot, parser and metadata manifests, citation-lineage graphs, normalized mappings, and semantic chunks as SHA-256-addressed objects. A content-addressed relation binds those objects to one snapshot identity. Publication serializes writers, persists and verifies the objects, relation, and immutable generation, and replaces active.json last. Readers consequently observe the prior complete generation until the new one is fully durable. Integrity-bound reuse bundles reconstruct prior typed documents after process restart. Ingest still rediscovers and read-admits the current directory, then reuses a derivation only when source bytes, format, parser contract, and the relevant configuration identity agree. Metadata is resolved again from current reviewed truth; chunk-policy changes rebuild chunks without reparsing content, while parser-contract changes reparse only the affected formats.

Repeating the same snapshot returns an explicit unchanged result and an empty identity-bound delta. Add, modification, delete, and rename transitions publish a new immutable generation with tombstones and affected chunk identities while leaving every prior generation byte-identical. Recovery may remove abandoned staging entries or restore previous.json; it never promotes staged content.

Cache Identity

DiskCache stores bytes; serialization belongs to the caller. Its filename identity includes namespace, caller-supplied version, and a SHA-256 key digest. Changing cleaning, chunking, embedding, or serialization meaning requires a new namespace or version.

content_hash_key normalizes chunk text and includes a normalization version before calculating a BLAKE2b digest. It identifies normalized content, not the source license, complete pipeline configuration, embedder, or corpus generation. Do not use it alone as a run or publication identity.

Recovery Decisions

Failure Safe response Unsafe response
interrupted JSONL publication keep the previous target; inspect or discard the temporary sibling concatenate partial and previous outputs
interrupted canonical snapshot publication remove abandoned staging and retain or restore the previously verified active manifest point active.json at a staged or partially validated generation
missing or corrupt snapshot CAS object/relation refuse the active generation and recover the prior verified generation trust a generation manifest without checking all reachable content
invalid MessagePack or fingerprint mismatch quarantine and rebuild from the recorded generation edit binary state or ignore the mismatch
stale cache semantics rotate namespace/version and recompute assume bytes are valid because the key exists
HTTP process restart rebuild or load durable index state assume an old index_id still resolves
mixed chunk and index generations reject the handoff and republish one generation select independently newest files

Use a caller-owned root such as artifacts/ingest/<run-id>/, restrict access to the source classification, and validate an index by loading and querying it before downstream publication.

See artifact contracts for durable fields and failure recovery for operator actions.