Storage API¶

Artifacts¶

Persist and reload every stage's output, keyed by document checksum, on the configured backend (s3 | local).

from ingestlib.storage import artifacts

ingestlib.storage.artifacts.save_parse ¶

save_parse(result: ParseResult) -> str

Persist a ParseResult and all its binary artifacts. Returns the doc_id.

The JSON carries every structural field (regions, bboxes, markdown, ...); page renders and figure crops are written as separate PNG objects.

ingestlib.storage.artifacts.load_parse ¶

load_parse(
    doc_id: str, *, include_images: bool = False
) -> ParseResult

Load a persisted ParseResult.

include_images=False (default) returns pages with image_bytes=None and figure crops as empty bytes — cheap, structure-only. include_images=True fetches every PNG back into the result.

ingestlib.storage.artifacts.load_classify ¶

load_classify(doc_id: str) -> ClassifyResult

Load a persisted ClassifyResult.

ingestlib.storage.artifacts.load_split ¶

load_split(doc_id: str) -> SplitResult

Load a persisted SplitResult.

ingestlib.storage.artifacts.save_extract ¶

save_extract(doc_id: str, result: ExtractResult) -> None

Persist an ExtractResult, keyed by its schema name — extractions with different schemas against the same document coexist.

ingestlib.storage.artifacts.load_extract ¶

load_extract(doc_id: str, schema: type) -> ExtractResult

Load a persisted ExtractResult for schema, revalidating every item's value back into the schema class (they round-trip as plain dicts).

ingestlib.storage.artifacts.load_ingest_manifest ¶

load_ingest_manifest(doc_id: str) -> dict[str, Any]

Load the vector-store sync record written by save_ingest_manifest.

ingestlib.storage.artifacts.document_exists ¶

document_exists(doc_id: str) -> bool

True when this document was parsed and saved before (dedup check).

ingestlib.storage.artifacts.ingest_complete ¶

ingest_complete(doc_id: str) -> bool

True when the FULL pipeline finished for this document.

Checks the ingest manifest — the last artifact the pipeline writes — so a run that died after parse/classify/split gets retried instead of skipped.

ingestlib.storage.artifacts.list_documents ¶

list_documents() -> list[DocumentMeta]

Registry of every persisted document — id, filename, pages, category, counts.

ingestlib.storage.artifacts.get_document_meta ¶

get_document_meta(doc_id: str) -> DocumentMeta

Registry entry for one document (self-healing, like list_documents).

ingestlib.storage.artifacts.page_image_key ¶

page_image_key(doc_id: str, page_num: int) -> str

Artifact key of a page render — read_blob() serves it on any backend (on s3, get_s3_client().generate_presigned_url can serve it as a URL).

ingestlib.storage.artifacts.read_blob ¶

read_blob(key: str) -> bytes

Raw bytes at an artifact key, whichever backend holds them.

The backend-agnostic way for a UI to serve page renders and figure crops when a presigned URL is not available (artifact_store: local).

ingestlib.storage.artifacts.delete_document ¶

delete_document(doc_id: str) -> int

Remove every object under the document's prefix. Returns count deleted.

The VectorStore contract¶

Every connector implements this interface — code written against it runs on any backend.

ingestlib.storage.base.VectorStore ¶

Bases: ABC

Contract for pushing split chunks into a vector database and querying them.

Implementations must make upserts idempotent per (document_id, chunk_id) — re-ingesting a document overwrites its vectors, never duplicates them.

upsert_chunks `abstractmethod` ¶

upsert_chunks(
    document_id: str,
    chunks: list[Chunk],
    embeddings: list[list[float]],
    category: str = "",
    namespace: str = "",
) -> int

Store one embedding per chunk with full provenance payload.

Returns the number of vectors written. embeddings[i] belongs to chunks[i]; use _validate_upsert() to enforce the pairing. category is the document-type label (from classify) stored on every vector so queries can filter by it.

query `abstractmethod` ¶

query(
    vector: list[float],
    top_k: int = 10,
    filters: dict[str, Any] | None = None,
    namespace: str = "",
    text: str | None = None,
) -> list[RetrievedChunk]

Nearest chunks to vector, best first.

filters are equality constraints on payload fields, e.g. {"category": "research_paper", "section": "methods"}. text is the original query text — connectors with a lexical/hybrid side use it for sparse search; dense-only connectors ignore it.

A hybrid connector that cannot fuse its two signals server-side (pinecone) may return MORE than top_k results — the dense top_k in order, then lexical-only extras — expecting a reranker to produce the final order; callers skipping reranking should slice [:top_k].

delete_document `abstractmethod` ¶

delete_document(
    document_id: str, namespace: str = ""
) -> int

Remove every vector belonging to a document. Returns count removed.

ingestlib.storage.base.RetrievedChunk ¶

Bases: BaseModel

One query hit — a stored chunk restored with its retrieval score.

Carries everything needed to answer AND cite: content (markdown/text), location (document_id, pages, region_ids → bboxes via the artifact store), and context (section, heading, category).

Connectors¶

from ingestlib.storage import (
    SqliteStore, PineconeStore, QdrantStore, PgvectorStore,
    MongodbStore, MilvusStore, OpensearchStore, WeaviateStore,
    default_store,
)

ingestlib.storage.default_store ¶

default_store() -> VectorStore

The connector selected by config.yaml's vector_store key.

All eight constructors take hybrid: bool = True — pass hybrid=False for dense-only behavior. Connection details come from configuration, never constructor arguments — see Connect a vector store.