CLI Surface¶
bijux-canon-ingest supports canonical corpus ingestion, retrieval operations,
and the configured document pipeline. If the first argument is corpus,
index, retrieve, ask, or eval, that operation parser owns the
invocation. Otherwise the command is interpreted as the configured document
pipeline.
flowchart LR
CLI[bijux-canon-ingest] --> Dispatch{first argument}
Dispatch -->|corpus| Corpus[canonical directory ingest]
Dispatch -->|index, retrieve, ask, eval| Retrieval[retrieval commands]
Dispatch -->|anything else| Pipeline[document pipeline]
Pipeline --> Chunks[chunk JSONL]
Retrieval --> Index[index artifact or query output]
Corpus --> Snapshot[immutable corpus snapshot]
Command Map¶
| Form | Required inputs | Primary output |
|---|---|---|
corpus build |
document root, logical root name, corpus name | canonical snapshot summary; optional atomic publication |
legacy-csv INPUT.csv --config CONFIG |
CSV and pipeline JSON | preserved validation/process outcome; optional chunk JSONL |
index build |
CSV and output path | MessagePack index and fingerprint summary |
retrieve |
index and query | ranked candidate JSON |
ask |
index and query | extractive answer with citations in JSON or YAML |
eval |
index and suite directory | recall-at-k metrics and regression status |
Build a Canonical Corpus¶
bijux-canon-ingest corpus build \
--root documents \
--root-name reviewed-documents \
--corpus-name reviewed-documents \
--publish-root artifacts/ingest/published
An adjacent corpus.lock.json is discovered automatically. Use
--corpus-lock /path/to/corpus.lock.json to select an explicit verified lock.
Lock and acquisition evidence is checked before parsing and retained in each
document's canonical metadata. If no lock is present, the command still works
with explicitly lower discovery/filename provenance. Invalid or contradictory
lock evidence exits with status 2 and includes its stable refusal code on
stderr. Successful JSON includes a portable corpus_lock summary with the
verified schema, identity, source count, and discovery mode, or status equal
to absent for an unlocked directory. With --publish-root, the result also
declares disposition as initial, unchanged, or changed. A restorable
prior generation supplies an exact delta; unchanged runs reuse every eligible
document after restart without invoking its parser, while changed runs name
the precise additions, modifications, deletions, renames, tombstones, and
chunk invalidations.
Directory traversal is bounded by default. Use --max-depth, --max-entries,
--max-files, --max-file-bytes, --max-total-bytes, and --max-seconds to
select stricter or explicitly reviewed limits. A limit refusal exits with
status 2 and its stable *_limit_exceeded code; it never publishes the files
seen before exhaustion as a complete snapshot. --symlink-policy defaults to
reject; the two within-root modes still reject escapes and cycles.
Preserved CSV Compatibility Pipeline¶
bijux-canon-ingest legacy-csv documents.csv \
--config pipeline.json \
--set chunk.chunk_size=384 \
--out artifacts/ingest/chunks.jsonl
--set is repeatable and accepts dotted key=value overrides. The command
loads all admissible documents, builds the configured step sequence, and writes
only successful chunks when --out is present. The JSONL file does not contain
error rows or a run manifest; retain the command outcome and effective
configuration separately. The historical path-first form remains an exact
alias during the 0.4 release line; new multi-format workflows should use
corpus build rather than converting documents to CSV.
Build an Index¶
bijux-canon-ingest index build \
--input documents.csv \
--out artifacts/ingest/corpus.index \
--backend bm25 \
--chunk-size 384 \
--overlap 48 \
--tail-policy emit_short
--backend accepts bm25 or numpy-cosine. Dense indexes accept hash16 or
the optional sbert embedder; --sbert-model selects the external model.
The command prints JSON containing the output path, fingerprint, and backend.
Retrieve and Answer¶
bijux-canon-ingest retrieve \
--index artifacts/ingest/corpus.index \
--query "retention period" \
--top-k 5 \
--filter category=governance \
--out artifacts/ingest/candidates.json
bijux-canon-ingest ask \
--index artifacts/ingest/corpus.index \
--query "How long are signed records retained?" \
--top-k 5 \
--format json \
--out artifacts/ingest/answer.json
--filter key=value is repeatable. ask reranks by default; use
--no-rerank to preserve the initial candidate order. YAML output requires the
optional PyYAML dependency.
Evaluate¶
bijux-canon-ingest eval \
--index artifacts/ingest/corpus.index \
--suite evaluation/retention \
--k 10 \
--baseline evaluation/retention/baseline.json \
--tolerance 0.01
The suite directory must contain queries.jsonl. Each usable row supplies a
query and relevant document IDs. The command returns status 1 when recall
falls below the baseline beyond tolerance and 2 when the suite is missing.
Automation Contract¶
Pipeline argument or parse failures use status 2; governed pipeline failures
use status 1; success uses status 0. Retrieval commands emit machine-readable
JSON for their principal results, but unexpected file, dependency, or payload
errors can still terminate through the command runtime. Scripts should capture
stderr, require the expected output file, and validate its schema before
publication.
See Artifact Contracts before treating JSONL or an index as a durable handoff.