Ingest Handbook

bijux-canon-ingest turns source documents into deterministic records, chunks, local retrieval indexes, ranked candidates, and extractive answers with citations. It supports both small in-process transformations and file-backed pipelines without making HTTP, CLI, or orchestration dependencies mandatory at package import time.

Ingest owns uncertainty in source shape. Invalid chunk geometry, malformed CSV rows, unsafe filtering rules, retry exhaustion, and circuit-breaker decisions remain explicit results rather than becoming silent changes to downstream evidence.

flowchart LR
    csv["CSV or RawDoc stream"]
    rules["safe rules + CleanConfig"]
    clean["CleanDoc"]
    chunks["ChunkWithoutEmbedding"]
    index["BM25 or NumPy cosine index"]
    result["candidates or cited answer"]

    csv --> rules --> clean --> chunks --> index --> result
    rules -. typed ErrInfo .-> result

Available Surfaces

Surface	Concrete operations	Stable evidence
Python root	RawDoc, CleanDoc, RagEnv, clean_doc, chunk_doc, streaming combinators, Result, retry and breaker helpers	__all__, API-freeze tests, typed marker
command	CSV pipeline; index build; retrieve; ask; eval	parser tests and end-to-end fixtures
HTTP v1	health, chunk, index build, retrieve, ask	apis/bijux-canon-ingest/v1/schema.yaml
storage	CSV document input, JSONL chunk output, persisted BM25 or NumPy-cosine index	adapter and round-trip tests

The package-local index and extractive-answer features support an ingest-owned workflow. bijux-canon-index remains the owner of declared vector execution, backend capability negotiation, provenance-rich execution artifacts, and replay comparison across vector backends.

Start With One Document

The dependency-light Python root exposes the smallest complete preparation path. It accepts a typed source, applies canonical cleaning, validates chunk geometry, and returns chunks with parent identity and offsets:

from bijux_canon_ingest import RagEnv, RawDoc, chunk_doc, clean_doc

source = RawDoc(
    doc_id="policy-17",
    title="Retention policy",
    abstract="  Keep signed run records for seven years.  ",
    categories="governance",
)
prepared = clean_doc(source)
chunks = chunk_doc(prepared, RagEnv(chunk_size=48, overlap=8))

for chunk in chunks:
    print(chunk.doc_id, chunk.start, chunk.end, chunk.text)

For this input, the single chunk retains policy-17, offsets 0..40, and the normalized text keep signed run records for seven years.. That result proves the behavior of this local transformation under the supplied configuration. It does not prove source accuracy, durable persistence, embedding identity, or runtime consumption.

When the source is a corpus rather than one object, use the configured CLI pipeline and retain the JSONL output together with the exact configuration and source identity. The transition from an in-memory chunk to a reviewable corpus is an artifact-custody decision, not merely a change of entrypoint.

Follow One Prepared Document

Boundary	Retained evidence	Review question
parse	source identifier, input fields, adapter result	were the intended bytes accepted without silent coercion?
clean	CleanConfig, normalized text, safeguard outcome	which rules changed the source and which content was refused?
chunk	chunk geometry, parent identity, offsets, chunk records	can every chunk be traced to its exact prepared parent?
persist	JSONL records or local index manifest	can a later process load the same prepared material?
retrieve	query, index identity, ranked candidates, citations	which ingest-local records produced this extractive answer?

The first reviewable result is not the answer text. It is the chain from source identity through configuration and chunk offsets to ranked records. The entrypoint examples show the Python, CSV, local-index, and HTTP forms of that chain.

Boundary With Index

Ingest owns preparation and its dependency-light BM25 or NumPy-cosine local workflow. Index owns declared vector execution across backend capabilities, budgets, provenance-rich execution artifacts, and replay comparison. Move the question to index when backend selection, approximation, vector execution, or cross-backend comparison becomes the disputed decision.

source bytes -> ingest records and chunks -> index execution request

Reason, agent, and runtime may consume ingest artifacts. They must not repair missing source identity, invent chunk provenance, or reinterpret a preparation failure as empty evidence.

Runtime Handoff Is Not Yet An Exported Adapter

The implemented ingest retrieval boundary is path-based: bijux_canon_ingest.application.retrieve opens a persisted index and returns typed candidates. Runtime currently requests a different package-root callable using query, scope, and vector-contract identity. No canonical root adapter currently reconciles those contracts.

This gap does not weaken the package-local ingest workflows; it limits the claim that runtime can execute them as a lower-layer step. A trustworthy adapter must bind the preparation receipt and index identity to runtime's evidence record instead of reducing the handoff to text and score.

The Preparation Receipt

Prepared text is admissible downstream only when the preparation decision can be reconstructed. Retain a receipt beside the prepared records with these identities:

Identity	What it answers	What is insufficient
source	which input object or byte set entered preparation?	a display name without a digest or stable source identifier
configuration	which cleaning rules, safeguards, and overrides ran?	the name of a configuration file whose contents can change
transformation	which normalization and filtering outcomes occurred?	only the final text
segmentation	which chunk geometry, offsets, and tail policy were applied?	chunk text without its prepared parent
output	which record set or persisted artifact was emitted?	a path that may be overwritten
failures	which inputs were rejected, retried, truncated, or omitted?	a successful-record count alone

This receipt is the custody transfer to index. Index may attach embeddings, backend parameters, rankings, and execution provenance to the prepared identities. It must not replace them. A digest match demonstrates identity of the retained material; it does not demonstrate that the original source was accurate or complete.

Evidence And Limits

Claim	Evidence to inspect	Limit
cleaning is deterministic	input identity, normalized configuration, output record, repeated serialization	does not prove source truth
a chunk is traceable	parent identity, offsets, text, stable record shape	depends on retaining the prepared parent
local retrieval is reproducible	index type and identity, corpus records, query, ranking output	applies to the ingest-local backend, not every vector backend
runtime consumed ingest retrieval	installed-package adapter test, retained preparation receipt, mapped evidence identities	package co-installation alone does not establish this handoff
an extractive answer is cited	candidate records and cited spans	does not establish that the corpus is complete
bulk processing handled failure honestly	policy, typed ErrInfo, stage context, error counts	collected errors still require caller disposition

Continue By Question

Question	Next page
what belongs inside the preparation boundary?	Foundation
how do processing, application, and adapters depend on one another?	Architecture
which Python, CLI, HTTP, and storage contracts are callable?	Interfaces
how do I install, run, diagnose, or recover a pipeline?	Operations
what evidence protects deterministic preparation?	Quality

Failure Boundaries

• parsing and configuration failures identify the invalid input or override
• transformation failures use typed ErrInfo values and retain stage context
• bulk processing can fail fast, collect errors, cap errors, or stop at an explicit error-rate threshold
• retries, circuit breakers, resource guards, and caches are separate policies; enabling one does not silently imply another
• optional YAML, Typer, NumPy, sentence-transformer, and HTTP integrations are loaded at their owning boundary rather than on a dependency-light root import