Architecture

Layered, strict downward dependencies — nothing lower knows what sits above it.

src/ingestlib/
├── services/       ingest · retrieve · lifecycle (remove · sync · reindex · recollect) · verify — the product
├── operations/     parse · classify · split · extract — the tools (each standalone)
├── storage/        registry (Postgres metadata) · artifacts (S3 | local, bytes) · VectorStore contract · 8 connectors
├── sources/        structured retrieval — SQL databases & the corpus as queryable Sources
├── foundations/    llm (Bedrock · OpenAI · Ollama · Jina rerank) · ocr (PaddleOCR-VL)
├── cli/            the `ingestlib` command — init · doctor · ingest · sync · list · show · collections · remove · reindex · recollect · verify · registry · search · describe-schema · eval-sql · mcp
├── mcp/            MCP server (ingestlib[mcp]) — the tools/services exposed to agents
├── utils/          logger · files · sync · aws
└── config.py       config.yaml + .env + rules.yaml + sources.yaml → typed, frozen configs

src/ingestlib_registry/   the registry's Postgres schema + Alembic migrations
                          (a standalone package ingestlib depends on, one way)

The load-bearing decisions

Provider dispatch is a per-call config read. Operations import chat/embed_text from one surface; which backend answers is decided at call time. No client is built until a call happens, and backends that aren't selected are never imported — a sqlite + ollama pipeline never touches boto3.

The VectorStore contract absorbs backend quirks. ID schemes, metadata encoding, fusion mechanics, deletion semantics — each connector handles its backend's reality internally (documented at the top of each module) so pipelines are written once. Shared guarantees: idempotent upserts, orphan pruning on re-ingest, no infrastructure creation on the read path, namespace isolation everywhere.

The registry is the source of truth; vectors are a derived index. The Postgres registry records every document's structure, chunks, extractions, and lifecycle; the artifact store keeps the bytes (source, page renders, crops, whole-doc markdown). So nothing about a corpus is ever locked inside a vector database — reindex() rebuilds a vector store straight from the registry, no re-parse.

Provenance is structural, not annotated. Chunks record the parse region ids they cover, chunk boundaries can't cut through a region, and the full payload rides on the vector. The citation chain (hit → regions → bboxes → page render) needs no extra database.

Generated SQL is bounded, never trusted. Structured retrieval lets the LLM write read-only SQL over your databases, but behind a permission boundary whose floor is a read-only database role — so a wrong query is a wrong read, never a write. A statement allowlist, an injected LIMIT, and a timeout are defense in depth on top; verified queries let you pin reviewed SQL for answers that must be exact. On a wide schema, the same layer retrieves the relevant tables (embedding each table into a card, then closing over the foreign-key graph so the fed sub-schema is join-complete) rather than dumping every table — schema-RAG, reusing the very same embedding surface as the verified match, no vector store. The sources/ layer reuses foundations/llm (generate, param-fill, semantic match) — no new model, the same dispatch as everything else.

Errors carry their fix. Every backend boundary translates its classic failures — wrong AWS profile, missing model access, dead Ollama, exhausted Jina quota — into one-sentence remedies, re-raised with the original chained. ingestlib doctor is those same translations, run proactively.

Config is discovered at call time, never at import. Importing ingestlib does nothing; the first real call finds config.yaml (explicit env var, else CWD and parents). Frozen dataclasses make a loaded config immutable; changing files mid-process takes an explicit reset.

Testing philosophy

Real APIs, never mocks. Pure logic runs on every test invocation; server-hitting suites are opt-in via RUN_*_E2E gates. Failures are tested by provoking real ones — bogus keys get real 401s, dead ports get real connection refusals — so the error translations are verified against reality, not against a mock's guess. The sqlite connector's full suite runs ungated: there is no server, so in-process is the real thing.