Storage API

The corpus lives in two stores that share the doc_id (content checksum) as their join key: the registry (Postgres) holds everything queryable — structure, classification, chunks, extractions, lifecycle — and the artifact store (s3 | local) holds the bytes.

Registry — read a stored document

The registry is authoritative for metadata. Read a whole document back — its structure from the registry, its bytes lazily from the artifact store.

from ingestlib.services import get_document

ingestlib.services.document.get_document

get_document(doc_id: str) -> StoredDocument | None

The live stored document for doc_id, or None if unknown / a tombstoned replacement (whose bytes are gone; the row survives only for lineage).

ingestlib.services.document.StoredDocument

Bases: BaseModel

One document as the registry knows it, with lazy access to its blobs.

pages/sections/chunks/regions/extractions are the registry rows (plain dicts). markdown() and page_image() fetch bytes from the blob store on demand — they are NOT loaded eagerly.

markdown

markdown() -> str | None

The whole-document markdown from the blob store (None if absent).

page_image

page_image(page_num: int) -> bytes

The rendered PNG for one page (raw bytes) from the blob store.

Artifacts — the bytes

The artifact store holds a document's bytes only: the source file, page renders, figure crops, and the whole-document markdown. Everything queryable lives in the registry.

from ingestlib.storage import artifacts

ingestlib.storage.artifacts.save_parse

save_parse(result: ParseResult) -> str

Persist a ParseResult's BYTES — source file, page renders, figure crops, and the whole-document markdown. Structure + metadata go to the registry.

ingestlib.storage.artifacts.load_split

load_split(doc_id: str) -> SplitResult

Load a document's split output, reconstructed from the registry.

ingestlib.storage.artifacts.document_markdown

document_markdown(doc_id: str) -> str | None

The whole-document markdown blob (parse/document.md), or None if absent.

ingestlib.storage.artifacts.page_image_key

page_image_key(doc_id: str, page_num: int) -> str

Artifact key of a page render — read_blob() serves it on any backend (on s3, get_s3_client().generate_presigned_url can serve it as a URL).

ingestlib.storage.artifacts.read_blob

read_blob(key: str) -> bytes

Raw bytes at an artifact key, whichever backend holds them.

The backend-agnostic way for a UI to serve page renders and figure crops when a presigned URL is not available (artifact_store: local).

ingestlib.storage.artifacts.missing_blobs

missing_blobs(doc_id: str, filename: str) -> list[str]

Essential blobs that should exist for a stored document but don't — the source bytes (the ultimate truth: the full pipeline is re-runnable from them) and the assembled document.md. The audit half of verify for the blob store.

ingestlib.storage.artifacts.document_exists

document_exists(doc_id: str) -> bool

True when this document has a registry row (dedup check).

ingestlib.storage.artifacts.ingest_complete

ingest_complete(doc_id: str) -> bool

True when the FULL pipeline finished (the vector/embed step ran).

ingestlib.storage.artifacts.list_documents

list_documents() -> list[DocumentMeta]

Every live document in the registry — id, filename, pages, category, counts.

ingestlib.storage.artifacts.get_document_meta

get_document_meta(doc_id: str) -> DocumentMeta

Registry entry for one document (empty meta when unknown).

ingestlib.storage.artifacts.find_by_path

find_by_path(
    path: Path | str, namespace: str = ""
) -> DocumentMeta | None

The live document currently claiming this source path — logical identity.

Matched on the resolved absolute path AND namespace via an indexed registry query (replaced/tombstoned rows excluded); newest created_at wins.

ingestlib.storage.artifacts.delete_document

delete_document(doc_id: str) -> int

Remove every object under the document's prefix. Returns count deleted.

The VectorStore contract

Every connector implements this interface — code written against it runs on any backend.

ingestlib.storage.base.VectorStore

Bases: ABC

Contract for pushing split chunks into a vector database and querying them.

Implementations must make upserts idempotent per (document_id, chunk_id) — re-ingesting a document overwrites its vectors, never duplicates them.

upsert_chunks abstractmethod

upsert_chunks(
    document_id: str,
    chunks: list[Chunk],
    embeddings: list[list[float]],
    category: str = "",
    namespace: str = "",
) -> int

Store one embedding per chunk with full provenance payload.

Returns the number of vectors the backend ACKNOWLEDGED writing — the durability signal the ingest ledger checks, so it must reflect what the store confirmed, not len(chunks) assumed. embeddings[i] belongs to chunks[i]; use _validate_upsert() to enforce the pairing. category is the document-type label (from classify) stored on every vector so queries can filter by it.

count_vectors abstractmethod

count_vectors(document_id: str, namespace: str = '') -> int

How many vectors the store currently holds for a document.

The audit half of the durability ledger: verify compares this live count against the registry's expected chunk_count to catch vectors that silently went missing (a partial upsert, an out-of-band deletion, a store rebuilt from a stale snapshot).

query abstractmethod

query(
    vector: list[float],
    top_k: int = 10,
    filters: dict[str, Any] | None = None,
    namespace: str = "",
    text: str | None = None,
) -> list[RetrievedChunk]

Nearest chunks to vector, best first.

filters are equality constraints on payload fields, e.g. {"category": "research_paper", "section": "methods"}. text is the original query text — connectors with a lexical/hybrid side use it for sparse search; dense-only connectors ignore it.

A hybrid connector that cannot fuse its two signals server-side (pinecone) may return MORE than top_k results — the dense top_k in order, then lexical-only extras — expecting a reranker to produce the final order; callers skipping reranking should slice [:top_k].

delete_document abstractmethod

delete_document(
    document_id: str, namespace: str = ""
) -> int

Remove every vector belonging to a document. Returns count removed.

ingestlib.storage.base.RetrievedChunk

Bases: BaseModel

One query hit — a stored chunk restored with its retrieval score.

Carries everything needed to answer AND cite: content (markdown/text), location (document_id, pages, region_ids → bboxes via the registry), and context (section, heading, category).

Connectors

from ingestlib.storage import (
    SqliteStore, PineconeStore, QdrantStore, PgvectorStore,
    MongodbStore, MilvusStore, OpensearchStore, WeaviateStore,
    default_store,
)

ingestlib.storage.default_store

default_store() -> VectorStore

The connector selected by config.yaml's vector_store key.

All eight constructors take hybrid: bool = True — pass hybrid=False for dense-only behavior. Connection details come from configuration, never constructor arguments — see Connect a vector store.