Module Map¶

bijux-canon-ingest owns the path from untrusted source records to stable, retrieval-ready chunks. Its architecture keeps pure document transformations separate from orchestration, optional adapters, and delivery interfaces so a caller can adopt only the boundary it needs.

flowchart LR
    A[Raw source records] --> B[core types and rules]
    B --> C[processing]
    C --> D[application workflows]
    D --> E[retrieval assembly]
    D --> F[interfaces]
    E --> F
    G[config] --> C
    H[result and safeguards] --> D
    I[infra and integrations] --> D
    J[observability] -. observes .-> C
    J -. observes .-> D
    F --> K[JSONL, index, or HTTP response]

The arrows describe dependency and data flow, not a requirement to use every layer. A library caller can use RawDoc, clean_doc, and chunk_doc without loading the CLI, FastAPI, storage, or embedding adapters.

Ownership by module¶

Module	Owns	Use it when
`core`	Immutable document and tree types, predicates, safe rule parsing, and structural deduplication	Defining source identity, selection rules, or ingest invariants
`processing`	Cleaning, chunk spans, chunk materialization, embedding boundaries, and deterministic stage composition	Transforming records directly or building a custom pipeline
`application`	End-to-end ingest, configured pipelines, indexing, evaluation, and service orchestration	Running a complete use case rather than one transform
`retrieval`	Ingest-local indexes, candidates, filters, reranking, citations, and artifact codecs	Building or querying the package's self-contained retrieval path
`interfaces`	Console commands, CSV/JSONL codecs, strict HTTP models, and the FastAPI v1 adapter	Crossing a process or network boundary
`infra` and `integrations`	Filesystem, vector, model, embedding, and other optional adapters	Connecting pure workflows to external capabilities
`result`	Typed success and failure values, collection, partitioning, and recovery	Making expected stage failures explicit in a stream
`safeguards`	Retry policy, circuit breakers, resource lifetimes, memoization, and error reports	Bounding operational risk around effectful work
`streaming`, `fp`, and `tree`	Lazy stream combinators, effect composition, and tree folds	Extending the execution model without hiding evaluation order
`observability`	Taps, traces, probes, and observation configuration	Recording behavior without changing stage results
`config`	Validated configuration objects and cleaner construction	Sharing stable configuration across callers

The transformation boundary¶

The central data progression is explicit:

RawDoc -> CleanDoc -> ChunkWithoutEmbedding -> Chunk

Cleaning normalizes the reader-visible text fields. Chunking attaches source identity and character offsets. Embedding is an effect boundary: external embedders can affect repeatability, while chunk identity is derived from stable document identity, offsets, and text. Observability taps may inspect these values but must not mutate or replace them.

Walk one source through the architecture¶

sequenceDiagram
    participant Edge as interfaces/readers
    participant Core as core
    participant Processing as processing
    participant App as application
    participant Retrieval as retrieval
    participant Obs as observability

    Edge->>Core: parsed fields + source identity
    Core-->>Edge: RawDoc or typed refusal
    App->>Processing: RawDoc + effective configuration
    Processing-->>Obs: stage observations
    Processing-->>App: CleanDoc + ordered chunks or typed failures
    App->>Retrieval: chunks + embedding/index specification
    Retrieval-->>App: artifact + candidates + citation records
    App-->>Edge: accepted/rejected inventory + outputs

The interface decodes and translates; it does not decide document identity or transformation semantics. application selects and orders stages; it does not reimplement cleaners or chunkers. retrieval consumes prepared material; it does not silently normalize or rechunk it. Every refusal returns through the same path with source and stage identity intact.

Dependency direction¶

From	May depend on	Must not acquire
`core`	identity and validation primitives	CLI, HTTP, storage, provider or pipeline orchestration
`processing`	core records, explicit configuration and effect protocols	caller paths, transport responses or hidden retry/cache policy
`application`	core, processing, results, safeguards and selected adapters	transformation semantics that belong in an owning stage
`retrieval`	prepared records, embedding/index contracts and codecs	reason-level claim support or governed multi-backend execution policy
`infra` / `integrations`	package protocols and external dependencies	authority to change core records or suppress typed failures
`interfaces`	application use cases, serialization and boundary errors	domain rules duplicated in CLI/HTTP handlers
`observability`	emitted values/events through taps and probes	permission to mutate the observed result

When a dependency would point upward—for example, a cleaner importing a CLI option model—move the behavior-bearing contract downward and keep adapter translation at the edge.

Where neighboring packages begin¶

Ingest can build a local BM25 or cosine index so a document pipeline can be used on its own. It does not own the governed execution contract of bijux-canon-index, evidence-backed reasoning in bijux-canon-reason, agent lifecycle policy, or runtime replay authority. Move to those packages when the problem changes from preparing source material to governing retrieval, reasoning, orchestration, or execution history.

Source and proof¶

processing contains the deterministic document stages.
application composes package workflows.
interfaces owns CLI, serialization, and HTTP edges.
tests exercises import isolation, transformations, adapters, and public contracts.