Skip to content

Connect a vector store

Three steps for any backend: install its extra (sqlite needs none — it ships with the core install), pick one key in config.yaml, put its connection secret in .env. Indexes, collections, tables, and search schemas are all created automatically on first ingest. All eight connectors deliver hybrid search; switching stores never changes your code.

vector_store: sqlite   # or any tab below

If you select a store whose SDK isn't installed, the error names the exact extra: pip install "ingestlib[qdrant]".

Prefer compose over the per-tab docker run commands? The repo's infra/docker-compose.yml starts any backend with one profile — docker compose -f infra/docker-compose.yml --profile qdrant up -d — with ports matching the tabs below, and encodes the startup gotchas (pg18 volume layout, mongodb's dual mounts, Milvus's companions).

The default. One local file, no server, no keys — sqlite-vec for dense KNN, FTS5 BM25 for lexical, fused with RRF.

vector_store: sqlite
# sqlite:
#   path: ingestlib.db      # relative paths anchor beside config.yaml

Nothing to install or run. Best single-machine choice.

Serverless, fully managed. Two indexes (dense + hosted sparse model) are created on first use.

uv add "ingestlib[pinecone]"

vector_store: pinecone

# .env — key from https://app.pinecone.io → API Keys
PINECONE_API_KEY=…

Local docker or Qdrant Cloud. Dense + BM25 sparse on one collection, fused server-side.

uv add "ingestlib[qdrant]"

vector_store: qdrant

docker run -p 6333:6333 qdrant/qdrant

# .env
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=            # cloud only

The Postgres you already run — HNSW cosine + built-in full-text over a generated tsvector. The extension is enabled and the table created automatically.

uv add "ingestlib[pgvector]"

vector_store: pgvector

docker run -p 5432:5432 -e POSTGRES_PASSWORD=pw pgvector/pgvector:pg18

# .env
PGVECTOR_URL=postgresql://postgres:pw@localhost:5432/postgres

Works with RDS, Supabase, Neon — anywhere the extension ships.

Atlas Vector Search + Atlas Search (true BM25). Atlas any tier, the atlas-local docker image, or self-managed 8.2+ with mongot.

uv add "ingestlib[mongodb]"

vector_store: mongodb

docker run -p 27017:27017 mongodb/mongodb-atlas-local

# .env
MONGODB_URL=mongodb://localhost:27017/?directConnection=true

Search indexes sync a few seconds behind writes — ingestion is unaffected; only write-then-query-immediately notices.

Dense ANN + server-computed BM25, fused server-side. Local standalone docker or Zilliz Cloud.

uv add "ingestlib[milvus]"

vector_store: milvus

# .env
MILVUS_URL=http://localhost:19530
MILVUS_TOKEN=              # Zilliz Cloud only

faiss k-NN + Lucene BM25. An Amazon OpenSearch domain — requests are SigV4-signed with your configured aws.profile, no separate key — or a local server.

uv add "ingestlib[opensearch]"

vector_store: opensearch

docker run -p 9200:9200 -e discovery.type=single-node \
  -e DISABLE_SECURITY_PLUGIN=true opensearchproject/opensearch

# .env
OPENSEARCH_URL=http://localhost:9200

For a real Amazon domain, the repo ships a CloudFormation template for the cheapest k-NN-capable configuration (~$0.10/hour, delete when idle): infra/opensearch.yaml.

HNSW dense + native BM25, fused server-side in one hybrid call. Local docker (publish both ports — the client speaks gRPC too) or Weaviate Cloud.

uv add "ingestlib[weaviate]"

vector_store: weaviate

docker run -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:latest

# .env
WEAVIATE_URL=http://localhost:8080
WEAVIATE_API_KEY=          # cloud only

Verify

uv run ingestlib doctor        # includes a reachability check for the selected store

Behaviors shared by every connector

  • Created on first use — no manual index/collection/table setup, and nothing is created on the read path
  • Idempotent re-ingestion — same document overwrites in place; a re-parse with fewer chunks prunes the leftovers
  • Namespaces and filters work identically everywhere
  • Index/collection/table names are configurable per backend — see the configuration reference
  • Only the selected connector's SDK is ever used at runtime

Migrating between stores

Moving stores is a rebuild, not an export: change vector_store, then run backfill() (or ingestlib backfill), which re-embeds every document straight from its stored split artifact — no re-parse — into the new connector.