CLI¶
Installed with the package as the ingestlib command
(uv run ingestlib …; python -m ingestlib is equivalent).
ingestlib --version prints the installed version.
| Command | Purpose |
|---|---|
init |
scaffold config.yaml (+ .env) |
doctor |
verify the configured stack with real calls |
ingest |
index files or folders |
sync |
reconcile a folder with the corpus |
list |
show every stored document |
show |
one document's full record (structure, chunks, extractions) |
collections |
the classify collections and their document counts |
remove |
erase a document (by path or doc_id) |
reindex |
rebuild the vector store from the registry |
recollect |
re-sort the corpus after editing classify rules |
verify |
audit durability (registry vs the vector & blob stores) |
registry |
manage the internal registry DB (init/status/backup/restore) |
search |
cited retrieval from the shell |
describe-schema |
auto-document a SQL source's tables (LLM-generated hints) |
eval-sql |
measure text2SQL accuracy on a source against a question set |
mcp |
serve the corpus to agents over MCP (needs ingestlib[mcp]) |
init and doctor set up and verify a stack; the rest manage the documents in
it — the corpus-management guide is Manage a corpus.
Most corpus commands take --namespace.
ingestlib init¶
Writes the setup files into the current directory — where config discovery looks.
| Flag | Effect |
|---|---|
| (none) | Default stack: Bedrock + sqlite + Jina. Writes config.yaml (with an aws: section to fill) and .env (with key slots). |
--local |
Zero-cloud stack: Ollama + sqlite + local artifacts + no reranker. Writes config.yaml only — no keys are needed, so no .env. |
--force |
Overwrite existing files. Without it, init refuses and exits 1 if config.yaml or .env already exists. |
Output ends with the preset's next steps (models to pull, keys to fill,
the OCR server command, and ingestlib doctor).
ingestlib doctor¶
Verifies the configured stack with real calls — a chat round-trip, an actual embedding, a store reachability probe. Checks only what your config.yaml choices require.
| Check | Failure mode |
|---|---|
| Python version | fail below 3.12 |
| Config discovery + parse | fail — and doctor stops; nothing else can run |
| LibreOffice | warn — DOCX/PPTX need it; PDF/images don't |
| OCR server | warn — parse/ingest need it; classify/split/extract/retrieve don't |
| LLM provider | fail — real chat call |
| Embedding provider | fail — real embedding; prints the dimension |
| Reranker | fail (skip when reranker: none) |
| Artifact store | fail — S3 credentials + bucket, or local-folder writability |
| Vector store | fail — liveness probe; never creates indexes or schema |
| Registry | fail — reachable and migrated (run ingestlib registry init) |
| Structured sources | fail — a SELECT 1 per declared source (skip when there's no sources.yaml) |
Marks: ✓ ok · ! warning · ✗ failure · - skipped.
Exit code 0 when nothing failed (warnings allowed), 1 otherwise —
safe to gate a deployment script on.
Every failure line carries the same one-sentence fix the library raises at runtime — the troubleshooting catalog lists them all.
Costs¶
Doctor's LLM and reranker probes are real calls: fractions of a cent on cloud providers, free on Ollama. Local checks (config, LibreOffice, sqlite, folders) are free.
ingestlib ingest¶
Index one or more files or folders. A folder expands to its ingestible files (PDF/DOCX/PPTX/PNG/JPEG/WebP, recursive).
ingestlib ingest report.pdf
ingestlib ingest docs/ contracts/ invoice.pdf # mix files and folders
ingestlib ingest report.pdf --namespace tenant-a
Prints a per-stage progress line and the outcome per file (ingested,
replaced, moved, or skipped — see Manage a corpus).
Exit 1 if any file failed; the rest still process.
ingestlib sync¶
Reconcile a folder with the corpus: new files ingest, edited files replace,
renamed files move, and — with --prune — deleted files are removed.
ingestlib sync corpus/ # ingest/replace/move
ingestlib sync corpus/ --dry-run # print the plan, change nothing
ingestlib sync corpus/ --prune # also delete gone files
ingestlib sync corpus/ --prune --dry-run # preview the prune first
| Flag | Effect |
|---|---|
--prune |
delete documents whose file is gone (root-scoped; refuses on an empty scan) |
--dry-run |
print the plan, execute nothing |
--namespace |
reconcile one partition |
Always --dry-run before a first --prune. Guardrails are detailed in
Pruning safely. Exit 1 if any
file errored.
ingestlib list¶
The registry as a table — doc_id (short), pages, chunks, category, namespace, source path.
ingestlib show¶
Print one document's full record from the registry — identity, category and confidence, page/section/chunk counts, and any persisted extractions.
ingestlib show report.pdf # by source path
ingestlib show 7b6b95d79149 # or by doc_id (full or a unique prefix)
ingestlib collections¶
List the classify collections and their document counts — the routing bins your
rules.yaml categories create, flagged for an attached extract schema /
auto-extract.
ingestlib remove¶
Erase one document from all three stores — vectors, then artifacts, then the registry row.
ingestlib remove report.pdf # by source path
ingestlib remove 7b6b95d79149 # by doc_id (full or a unique prefix)
Exit 1 when the target matches nothing (or a prefix is ambiguous) — nothing
is deleted.
ingestlib reindex¶
Rebuild the vector store from the registry — no re-parse. For a provider switch, a new store connector, or a wiped index (see reindex).
ingestlib recollect¶
Re-sort the corpus after editing your classify rules — re-classifies every stored document from the registry (no OCR, no re-parse), updating categories and collections in place (see recollect).
ingestlib verify¶
Audit the corpus against the registry — expected chunk counts vs what the vector
store actually holds, plus the essential blobs. --repair re-embeds vector-drifted
documents from the registry.
ingestlib registry¶
Manage the internal registry DB (Postgres) — ingestlib's metadata hub.
ingestlib registry init # create / upgrade the schema (Alembic)
ingestlib registry status # current revision + document count
ingestlib registry backup # pg_dump → the artifact store
ingestlib registry restore <key> # DESTRUCTIVE: pg_restore --clean from a backup
The registry runs from the bundled compose file —
docker compose -f infra/docker-compose.yml --profile registry up -d — then
ingestlib registry init applies the migrations.
ingestlib search¶
Cited retrieval from the shell — the ingest→verify loop without a Python session.
ingestlib search "what were the risks?"
ingestlib search "revenue growth" --top-k 10
ingestlib search "invoice INV-20114" --no-rerank
ingestlib search "how many are ready?" --sources prescriptions,rx_db
Prints ranked hits with score, citation, heading, and a snippet.
| Flag | Effect |
|---|---|
--top-k |
how many hits (default 5) |
--no-rerank |
skip the reranker (vector order) |
--sources |
comma-separated sources.yaml names to query (documents and/or SQL databases) — prints normalized results instead of document hits (structured retrieval) |
--namespace |
scope to one corpus partition |
ingestlib describe-schema¶
Auto-document a SQL source's tables for structured
retrieval.
Samples each table, asks the LLM for a one-line description, and prints a
ready-to-paste tables: block — it never edits sources.yaml, so you review
before trusting (a wrong hint silently misleads generation).
ingestlib describe-schema prescriptions # print to stdout
ingestlib describe-schema prescriptions --out hints.yaml
| Flag | Effect |
|---|---|
--out |
write the tables: block to a file (default: stdout) |
ingestlib eval-sql¶
Measure text2SQL accuracy on your own schema — the generate path is
analyst-assist, so measure how far to trust it. Runs a {question, expect}
set through the real answer() path and reports the overall match rate, the
generated-only rate (the honest production number), and every miss with the
SQL that ran.
ingestlib eval-sql prescriptions # <source>_eval.yaml beside sources.yaml
ingestlib eval-sql prescriptions --dataset questions.yaml
| Flag | Effect |
|---|---|
--dataset |
question/expected YAML (default: <source>_eval.yaml beside sources.yaml) |
The dataset is a list under questions: of {question, expect} — expect is a
value or list; a run hits when any expected value appears in the result. See
evals/sql_dataset.yaml
for the format. Like the retrieval eval, it measures — it never asserts.
ingestlib mcp¶
Serve the corpus to MCP agents (Claude Desktop, Cursor, …) as tools — the read
tools (search, extract, classify, list_documents, get_document,
collections, describe_schema, verify, doctor, registry_status) plus the
write tools (ingest, sync, remove, reindex, recollect, and registry
init/backup/restore). Needs the extra: pip install "ingestlib[mcp]".
Full guide: Serve to agents (MCP).
ingestlib mcp # stdio (local clients)
ingestlib mcp --transport http --port 8000 # remote; needs MCP_TOKEN
ingestlib mcp --read-only # only the read tools
| Flag | Effect |
|---|---|
--transport |
stdio (default, local) or http (streamable, remote) |
--host |
http bind address (default 127.0.0.1) |
--port |
http port (default 8000) |
--read-only |
hide the write tools (ingest/remove/sync/reindex/recollect + registry writes) |
The http transport requires MCP_TOKEN in .env (bearer auth) and binds
localhost unless you pass --host. stdio needs no token.