Storage¶
The vector-store contract, its eight connectors, and the artifact store.
The contract¶
ingestlib.storage.base.VectorStore ¶
Bases: ABC
Contract for pushing split chunks into a vector database and querying them.
Implementations must make upserts idempotent per (document_id, chunk_id) — re-ingesting a document overwrites its vectors, never duplicates them.
upsert_chunks
abstractmethod
¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Store one embedding per chunk with full provenance payload.
Returns the number of vectors written. embeddings[i] belongs to
chunks[i]; use _validate_upsert() to enforce the pairing. category
is the document-type label (from classify) stored on every vector so
queries can filter by it.
query
abstractmethod
¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks to vector, best first.
filters are equality constraints on payload fields, e.g.
{"category": "research_paper", "section": "methods"}.
text is the original query text — connectors with a lexical/hybrid
side use it for sparse search; dense-only connectors ignore it.
A hybrid connector that cannot fuse its two signals server-side (pinecone) may return MORE than top_k results — the dense top_k in order, then lexical-only extras — expecting a reranker to produce the final order; callers skipping reranking should slice [:top_k].
delete_document
abstractmethod
¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Remove every vector belonging to a document. Returns count removed.
ingestlib.storage.base.RetrievedChunk ¶
Bases: BaseModel
One query hit — a stored chunk restored with its retrieval score.
Carries everything needed to answer AND cite: content (markdown/text), location (document_id, pages, region_ids → bboxes via the artifact store), and context (section, heading, category).
ingestlib.storage.default_store ¶
default_store() -> VectorStore
The connector selected by config.yaml's vector_store key.
Connectors¶
Every connector implements the same contract; pick one with the
vector_store key in config.yaml or instantiate directly. See the
vector stores guide for choosing.
ingestlib.storage.sqlite.SqliteStore ¶
Bases: VectorStore
Vector storage in a local SQLite file (schema auto-created on first use).
hybrid=True (default) runs FTS5 BM25 next to every dense query and fuses both signals; hybrid=False is dense-only.
upsert_chunks ¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Store one row per chunk across all three tables in ONE transaction.
Returns the chunk count; the document's previous rows are replaced atomically, so re-ingestion overwrites, never duplicates.
query ¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks, best first; filters are equality constraints.
When hybrid and text is given, a BM25 branch runs next to the KNN
and both fuse with RRF — scores are then RRF ranks, not cosine.
delete_document ¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Delete the document's rows from all three tables. Returns count removed.
ingestlib.storage.pinecone.PineconeStore ¶
Bases: VectorStore
Vector storage on Pinecone serverless indexes (auto-created on first use).
hybrid=True (default) maintains the sparse lexical index alongside the dense one; hybrid=False is dense-only, exactly the v1 behavior.
upsert_chunks ¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Store one dense vector per chunk (plus its sparse twin when hybrid).
Returns the dense vector count; IDs make re-ingestion overwrite in place.
query ¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks, best first; filters are payload equality constraints.
When hybrid and text is given, the sparse index is searched with the
same top_k and its extra hits are appended after the dense results.
delete_document ¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Remove the document's vectors from both indexes.
Returns the dense count (the sparse index mirrors it 1:1, minus chunks that had no sparse form).
ingestlib.storage.qdrant.QdrantStore ¶
Bases: VectorStore
Vector storage on a Qdrant collection (auto-created on first use).
hybrid=True (default) stores a BM25 sparse vector next to every dense one and fuses both signals at query time; hybrid=False is dense-only.
upsert_chunks ¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Store one point per chunk (dense + sparse when hybrid).
Returns the point count; deterministic IDs overwrite in place.
query ¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks, best first; filters are payload equality constraints.
When hybrid and text is given, dense and BM25 branches run in one
server call fused with RRF — scores are then RRF ranks, not cosine.
delete_document ¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Delete the document's points by filter. Returns count removed.
ingestlib.storage.pgvector.PgvectorStore ¶
Bases: VectorStore
Vector storage in a Postgres table (extension + schema auto-managed).
hybrid=True (default) runs full-text search next to every dense query and fuses both signals; hybrid=False is dense-only.
upsert_chunks ¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Replace the document's rows in ONE transaction.
Returns the chunk count; delete-then-insert guarantees a re-parsed document with fewer chunks leaves no orphans behind.
query ¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks, best first; filters are equality constraints.
When hybrid and text is given, a full-text branch runs next to the
KNN and both fuse with RRF — scores are then RRF ranks, not cosine.
delete_document ¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Delete the document's rows. Returns the exact count removed.
ingestlib.storage.mongodb.MongodbStore ¶
Bases: VectorStore
Vector storage on a MongoDB collection (search indexes auto-created).
hybrid=True (default) runs BM25 $search next to every $vectorSearch and fuses both signals; hybrid=False is dense-only.
upsert_chunks ¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Replace the document's chunks (delete old rows, insert new ones).
Returns the chunk count. Deterministic _ids keep the operation idempotent, and the delete pass drops orphaned chunk_ids when a re-parse yields fewer chunks.
query ¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks, best first; filters are equality constraints.
When hybrid and text is given, a BM25 $search branch runs next to
$vectorSearch and both fuse with RRF — scores are then RRF ranks.
delete_document ¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Delete the document's chunks. Returns the exact count removed.
ingestlib.storage.milvus.MilvusStore ¶
Bases: VectorStore
Vector storage on a Milvus collection (auto-created on first use).
hybrid=True (default) runs the server's BM25 next to every dense search, fused server-side with RRF; hybrid=False is dense-only.
upsert_chunks ¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Replace the document's rows (delete old, insert new — raw text in, sparse vectors computed server-side).
Returns the chunk count; deterministic ids plus the delete pass keep re-ingestion idempotent and orphan-free.
query ¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks, best first; filters are equality constraints.
When hybrid and text is given, dense and BM25 branches run in one
server call fused with RRF — scores are then RRF ranks, not cosine.
delete_document ¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Delete the document's rows by filter. Returns count removed.
ingestlib.storage.opensearch.OpensearchStore ¶
Bases: VectorStore
Vector storage on an OpenSearch index (auto-created on first use).
hybrid=True (default) runs BM25 next to every k-NN search and fuses both signals; hybrid=False is dense-only.
upsert_chunks ¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Replace the document's rows (delete old, bulk-insert new).
Returns the chunk count. Deterministic _ids keep the operation idempotent, and the delete pass drops orphaned chunk_ids when a re-parse yields fewer chunks.
query ¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks, best first; filters are equality constraints.
When hybrid and text is given, a BM25 branch runs next to the k-NN
and both fuse with RRF — scores are then RRF ranks, not similarities.
delete_document ¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Delete the document's rows. Returns the exact count removed.
ingestlib.storage.weaviate.WeaviateStore ¶
Bases: VectorStore
Vector storage on a Weaviate collection (auto-created on first use).
hybrid=True (default) runs the server's BM25 next to every dense search, fused server-side; hybrid=False is dense-only.
upsert_chunks ¶
upsert_chunks(
document_id: str,
chunks: list[Chunk],
embeddings: list[list[float]],
category: str = "",
namespace: str = "",
) -> int
Replace the document's objects (delete old, insert new).
Returns the chunk count; deterministic object IDs plus the delete pass keep re-ingestion idempotent and orphan-free.
query ¶
query(
vector: list[float],
top_k: int = 10,
filters: dict[str, Any] | None = None,
namespace: str = "",
text: str | None = None,
) -> list[RetrievedChunk]
Nearest chunks, best first; filters are equality constraints.
When hybrid and text is given, dense and BM25 branches run in one
server call with ranked fusion — scores are then fused ranks, not
cosine.
delete_document ¶
delete_document(
document_id: str, namespace: str = ""
) -> int
Delete the document's objects by filter. Returns count removed.
Artifact store¶
ingestlib.storage.artifacts ¶
Artifact store — persists every operation's output, keyed by document checksum.
Lives on the backend artifact_store selects in config.yaml: an S3 bucket
(durable, shareable) or a plain local folder (zero cloud). Same layout on
both, everything under one prefix per document:
documents/{doc_id}/
├── source/{filename} original file, exact bytes
├── parse/result.json ParseResult (image bytes stripped)
├── parse/document.md whole-document markdown
├── parse/pages/page_0001.png ... page renders
├── parse/figures/{fig.filename} ... figure/chart crops
├── classify/result.json ClassifyResult
├── split/result.json SplitResult (chunks with provenance)
└── split/ingest_manifest.json vector-store sync record
doc_id is the parse checksum, so re-saving the same file overwrites in place and "already ingested?" is a single existence check. The citation chain needs no database: a vector hit's {doc_id, pages, region_ids} resolves to page images and bboxes straight from this layout.
DocumentMeta ¶
Bases: BaseModel
Lightweight per-document registry entry (stored as meta.json).
Written at save_parse; category / section counts are patched in by save_classify and save_split. Self-heals from parse/result.json when a document predates this file.
save_parse ¶
save_parse(result: ParseResult) -> str
Persist a ParseResult and all its binary artifacts. Returns the doc_id.
The JSON carries every structural field (regions, bboxes, markdown, ...); page renders and figure crops are written as separate PNG objects.
load_parse ¶
load_parse(
doc_id: str, *, include_images: bool = False
) -> ParseResult
Load a persisted ParseResult.
include_images=False (default) returns pages with image_bytes=None and figure crops as empty bytes — cheap, structure-only. include_images=True fetches every PNG back into the result.
save_classify ¶
save_classify(doc_id: str, result: ClassifyResult) -> None
Persist a ClassifyResult under the document's prefix.
save_split ¶
save_split(doc_id: str, result: SplitResult) -> None
Persist a SplitResult (sections + chunks with full provenance).
save_ingest_manifest ¶
save_ingest_manifest(
doc_id: str, manifest: dict[str, Any]
) -> None
Record what was pushed to the vector store (index, namespace, vector IDs).
load_ingest_manifest ¶
load_ingest_manifest(doc_id: str) -> dict[str, Any]
Load the vector-store sync record written by save_ingest_manifest.
document_exists ¶
document_exists(doc_id: str) -> bool
True when this document was parsed and saved before (dedup check).
ingest_complete ¶
ingest_complete(doc_id: str) -> bool
True when the FULL pipeline finished for this document.
Checks the ingest manifest — the last artifact the pipeline writes — so a run that died after parse/classify/split gets retried instead of skipped.
get_document_meta ¶
get_document_meta(doc_id: str) -> DocumentMeta
Registry entry for one document (self-healing, like list_documents).
list_documents ¶
list_documents() -> list[DocumentMeta]
Registry of every persisted document — id, filename, pages, category, counts.
page_image_key ¶
page_image_key(doc_id: str, page_num: int) -> str
Artifact key of a page render — presign it (s3) or read_blob() it (local).
read_blob ¶
read_blob(key: str) -> bytes
Raw bytes at an artifact key, whichever backend holds them.
The backend-agnostic way for a UI to serve page renders and figure crops when a presigned URL is not available (artifact_store: local).
delete_document ¶
delete_document(doc_id: str) -> int
Remove every object under the document's prefix. Returns count deleted.