01
Bring your sources together
Crawl websites with HTTP or a real browser, and acquire public or token-authenticated Git repositories through one durable pipeline.
Web + Git ingestion
Open-source RAG platform
Turn your sources into trusted context.
ContextCrate is an open-source, self-hosted RAG platform that ingests your knowledge, retrieves what matters, and delivers grounded AI answers with evidence you can inspect.
Self-hosted by design No conversation storage
WebsiteDocs & pages
Git repositoryMarkdown & text
ContextCrateParse · chunk · extract
Hybrid retrievalBM25 + vectors + RRF
Verified answerGrounded with citations
What is retrieval-augmented generation?
Give AI the right context before it answers.
RAG—retrieval-augmented generation—connects a large language model to knowledge you control. Instead of relying only on what the model learned during training, it retrieves relevant source material for every question and uses that evidence to produce the answer.
That makes RAG important for real products: knowledge stays current, private information can remain on your infrastructure, and users can verify claims against the original sources. ContextCrate provides the complete open-source RAG pipeline—from ingestion and indexing to hybrid retrieval, cited generation, grading, and answer verification.
More relevantAnswers use the best available context
TraceableCitations lead back to source evidence
Under controlSelf-host your context and indexes
QUESTION
“What is our current backup policy?”
Without RAGUNSUPPORTED
LLM→
A plausible answer based on old or generic training data.
VS
With ContextCrateGROUNDED
010203
→“Backups run daily with checksummed exports.” [1]
1
Operations handbook/docs/backup-policy · retrieved now
Built for trustworthy retrieval
Everything between your data and the answer.
One durable system for acquisition, transformation, retrieval, and grounded generation—without stitching together a maze of services.
02
Isolate every context space
Each crate owns its sources, members, artifacts, credentials, retrieval policy, and index namespace—without leaking context across teams.
Role-based access
03
Retrieve with precision
Combine BM25 lexical search and semantic vectors with reciprocal-rank fusion, then refine results with local or remote rerankers.
Hybrid search
04
Index meaning, not noise
Optional proposition retrieval generates independently useful facts, grades their quality, and preserves the original source behind every match.
Proposition retrieval
05
Answer with evidence
Stream cited answers from any OpenAI-compatible endpoint, grade retrieved context, and verify that generated claims stay grounded.
Citations + verification
06
Evolve without downtime
Rebuild versioned indexes in the background, activate them atomically, and move crates with checksummed export and import bundles.
Resilient operations
From source to supported answer
A pipeline you can reason about.
Every stage is explicit, observable, and scoped to the crate that owns it.
01
Connect
Create a crate, add a website or Git source, and define exactly what belongs in the ingestion job.
02
Structure
ContextCrate acquires raw artifacts, normalizes documents, creates deterministic chunks, and extracts metadata.
03
Retrieve
Search lexically, semantically, or in hybrid mode—optionally adding proposition retrieval and cross-encoder reranking.
04
Ground
Generate an answer from selected evidence, expose stable citations, and verify factual support before returning it.
Architecture without lock-in
Read the architecture docs ↗
Start in one process. Scale every stage.
The same domain model and work contracts power a compact standalone install and independently scalable distributed roles.
One canonical storeNormalized content remains the rebuildable source of truth.
Crate-aware everywhereIdentity travels with rows, artifacts, queue messages, and indexes.
Swappable infrastructureUse filesystem or S3, Lucene or OpenSearch, local or hosted models.
SYSTEM TOPOLOGY
WEBWebsite source
GITGit source
COREControl plane Auth · configuration · API
01Parser
02Extractor
03Indexer
DBRelational
OBJArtifacts
IDXSearch index
Deploy your way
A clean path from laptop to cluster.
Keep the operational footprint proportional to the workload.
START HERE
Standalone
Run the complete pipeline in one JVM with file-backed H2, filesystem artifacts, and Lucene.
terminal
$ docker run -p 8080:8080 \
ghcr.io/wenisch-tech/contextcrate:latest
- Fast local setup
- Persistent data volume
- Moderate crawl workloads
Distributed
Scale acquisition, parsing, and indexing roles independently across production infrastructure.
PostgreSQLRabbitMQS3OpenSearchKubernetesKEDA
- Horizontally scalable workers
- Queue-aware autoscaling
- Helm deployment included
Planned, not yet released
What comes next.
ContextCrate’s roadmap focuses on measuring retrieval quality, expanding ingestion, and making relevance improve safely over time.
Evaluation
Feedback & quality signals
Capture privacy-aware impressions, clicks, conversions, dwell time, and explicit feedback.
PLANNED
Connectors
Richer Git synchronization
Add incremental commits, webhooks, schedules, SSH acquisition, and deletion reconciliation.
PLANNED
Formats
Broader source coverage
Expand normalization beyond today’s web pages, Markdown, and UTF-8 text sources.
PLANNED
Ranking
Measured optimization
Build reproducible evaluation sets, ranking dashboards, and controlled learning-to-rank rollouts.
PLANNEDYour context. Your infrastructure.Build answers people can verify.
Explore the source, deploy ContextCrate, and turn your knowledge into context that earns trust.