Applied AI & Retrieval
Grounded Answer Engine
A question-answering system over a company’s own documents that cites every claim to a source span, respects the permissions of the person asking, and refuses when the evidence is not there.
At a glance
- Type
- Reference architecture
- Domain
- Applied AI & Retrieval
- Architecture
- 6 layers
- First production cut
- 9–13 weeks, phases in sequence
A reference architecture. Figures on this page are design targets, not measured client results.
The challenge
Every organisation has the answer written down somewhere — in a policy PDF, a Confluence page, a contract annex, a support macro from 2023 — and nobody can find it. The demand is not for a chatbot; it is for an answer a compliance officer can defend, with the paragraph it came from and a clear refusal when the corpus is silent. That means the hard parts are retrieval quality, permission fidelity and abstention, not the model.
Failure mode
The obvious approach, and why it breaks.
Almost every team reaches the same first design, and it is not a bad instinct — it is the shortest route to something that demos. Here is that design, and the point at which real volume takes it apart.
Discarded
The obvious approach
Dump every PDF into a vector database, split on 1,000 characters, embed with a single off-the-shelf model, retrieve top-k=5 by cosine similarity, paste the chunks into a prompt and ask the model to answer.
Why it breaks
What production does to it
It demos beautifully on twenty documents and collapses on twenty thousand. Dense-only retrieval misses exact-match queries — part numbers, clause references, error codes, internal acronyms — because embeddings smooth away the rare tokens that carry the question. Fixed-length splitting cuts tables in half and severs headings from the rows they govern, so a chunk arrives without the context that made it true. Everyone sees every document, because permissions were never part of the index. And with no eval set, nobody can tell whether the answer was grounded or fluent — the failure is silent, confident and unmeasurable, and the first time it surfaces is in front of a customer.
Architecture
How the system is put together.
6 layers, from the edge where work arrives to the operations that keep it honest. Each one names the components it owns and why it exists as a separate concern.
Layer 01
Ingestion & document understanding
Parse to structure before anything else. A document that arrives as a flat string has already lost the headings, table boundaries and page anchors that citations and chunking both depend on. Ingestion is also where permissions are captured — retrofitting them later means a full reindex.
Components
- Source connectors: SharePoint, Confluence, Google Drive, S3, Git, Zendesk
- Layout-aware parsing (Docling / Unstructured) to headings, tables, lists
- OCR fallback for scanned and image-only PDFs (Textract, Tesseract)
- Content-hash change detection and incremental re-ingestion
- ACL capture at source: groups, roles, tenant and document sensitivity
- Durable ingest workflows with retries and dead-letter inspection
Layer 02
Chunking & indexing
Two indexes over the same chunk identity. No transaction spans both, so writes go through a Postgres outbox with a reconciliation job that diffs chunk IDs and alerts on drift. Chunk IDs are stable across re-ingestion of an unchanged document, which is what makes incremental updates and citation permalinks possible.
Components
- Structure-driven splitting on heading hierarchy with token ceilings
- Parent–child chunks: retrieve the small span, expand to the section
- Contextual prefixes — document title and heading path prepended per chunk
- Lexical index (OpenSearch BM25) with analysers tuned for domain vocabulary
- Dense index (pgvector or Qdrant) with versioned embedding model tags
- Metadata columns: source, effective date, jurisdiction, ACL tuples, revision
Layer 03
Retrieval & ranking
This is where answer quality is actually decided. Fusion covers the complementary blind spots — BM25 for rare exact tokens, dense for paraphrase — and the reranker, which reads query and passage together, buys more accuracy than any prompt change. Permissions are a filter, never a post-hoc removal.
Components
- Query analysis: rewrite, decompose multi-part questions, expand acronyms
- Hard metadata pre-filter — tenant, ACL, effective date — applied inside the query
- Parallel BM25 and dense retrieval, k≈50 each, fused with reciprocal rank fusion
- Cross-encoder reranking (bge-reranker-v2-m3 or Cohere Rerank) down to 6–10 spans
- Maximal marginal relevance to stop one document flooding the context
- Score floor that returns an empty candidate set rather than weak matches
Layer 04
Grounding, citation & abstention
Citations are produced as structure, not as text the model was asked to format politely. An answer whose citation does not resolve to a real span is treated as a failed generation and either retried with tighter instructions or converted into a refusal.
Components
- Numbered evidence blocks with stable span offsets in the prompt
- Structured output: claims, each bound to one or more span IDs
- Span alignment check — every cited offset must exist and support the claim
- Groundedness verifier pass over claim–evidence pairs
- Abstention policy driven by fusion scores, rerank margin and verifier result
- Conflict surfacing when two in-date sources disagree
Layer 05
Evaluation & release gating
The eval set is the asset. Without it a model upgrade is a coin flip: nobody can say whether the new version improved answers or quietly started hallucinating on the narrow slice of queries that matter most. Every change ships behind a threshold, and a regression blocks the merge.
Components
- Golden set of real questions with labelled relevant documents and answers
- Retrieval metrics: recall@k, nDCG@10, MRR, permission-leak count (must be zero)
- Answer metrics: groundedness, citation precision, abstention correctness
- Adversarial slice: unanswerable, out-of-scope, stale-document and near-miss queries
- Harness in CI (Ragas, promptfoo) run on every prompt, chunker or model change
- Shadow replay of production traffic before any index or model cutover
Layer 06
Serving, cost & operations
Cost and trust are the same problem. Every trace stores the exact candidate set and scores behind an answer, so a disputed response can be reconstructed months later instead of argued about from memory.
Components
- Exact and semantic response caching with per-tenant, ACL-aware cache keys
- Provider prompt caching for the stable system and policy preamble
- Model tiering — small model for routing and rewriting, frontier model for synthesis
- Token budgets and truncation policy enforced before the call, not after the bill
- Tracing with OpenTelemetry and Langfuse: retrieved IDs, scores, tokens, latency
- Thumbs-down capture that writes straight into the eval backlog
Engineering decisions
Six decisions, and what each one costs.
A decision without a stated cost is marketing. These are the choices that shape the system, the reasoning behind them, and what is given up in exchange.
- 01
Hybrid lexical + dense retrieval fused with reciprocal rank fusion, not dense alone.
Why
Enterprise queries are full of tokens embeddings handle badly — SKU codes, clause numbers, internal project names, error strings. BM25 finds those exactly; dense retrieval finds the paraphrase. RRF combines the rankings without needing calibrated scores across two very different scales.
What it costs
Two indexes to build, back up, monitor and keep transactionally consistent, roughly double the write path, and one more tuning surface in the fusion constant. A single vector store would be materially cheaper to operate.
- 02
Retrieve wide, then rerank with a cross-encoder before the answer model sees anything.
Why
Bi-encoder similarity is a cheap approximation; a cross-encoder reads the query and passage jointly and reorders the candidate set far more accurately. Pulling k≈50 and cutting to 8 after reranking consistently beats pulling 8 directly.
What it costs
Adds roughly 100–300 ms and a GPU or per-call reranker cost on every query, and unlike embeddings the result cannot be precomputed — each query pays in full. Under heavy concurrency the reranker becomes the first thing to queue.
- 03
Permissions are enforced as a pre-filter inside the retrieval query, never by removing results afterwards.
Why
Post-filtering silently degrades quality — a user entitled to few documents gets a top-k that is mostly discarded — and one missed filter path leaks confidential text into a generated answer, where it is unrecoverable. Filtering inside the query makes the entitlement part of the retrieval contract.
What it costs
ACL changes now require reindexing the affected documents, which introduces a staleness window measured in minutes. Filter selectivity also hurts ANN recall, so the candidate pool has to be widened for narrowly entitled users, costing latency exactly where it is least welcome.
- 04
Chunk on document structure with parent–child expansion instead of fixed-size windows.
Why
Precision comes from indexing a small, specific span; sufficiency comes from giving the model the surrounding section. Splitting on the heading tree keeps tables and their captions together and gives every chunk a heading path that makes it interpretable on its own.
What it costs
The chunker becomes format-specific and needs real work per source type, and it degrades on scanned or badly authored PDFs where the heading tree is a lie. A naive splitter is one line of code and never crashes; this one needs a fallback path and its own tests.
- 05
Abstention is a first-class, measured outcome, gated by a groundedness verifier.
Why
For policy, legal and clinical questions a confident wrong answer costs far more than a refusal. Scoring the claim–evidence pairs before returning them converts hallucination from an invisible failure into a caught one, and lets the threshold be set per corpus.
What it costs
Coverage drops — the system says it cannot answer on questions where a looser configuration would have been right — and the verifier adds a second model call to the critical path. Threshold tuning is a genuine product decision, not a default, and it has to be revisited as the corpus grows.
- 06
Pin the embedding and generation model versions; upgrades ship only through the eval gate.
Why
A new embedding model changes the geometry of the entire index and a new generation model changes citation behaviour. Pinning makes those changes deliberate events with measured before-and-after numbers rather than a quiet drift nobody attributes correctly.
What it costs
The system deliberately runs behind the frontier, and a genuine improvement waits for a full reindex plus a dual-index migration window. On a large corpus that is real compute and real calendar time spent on a change users will never see.
Design targets
The operating point we build toward.
Targets, with the basis for each one written underneath it. They describe what this architecture is designed to hold — not a result measured on somebody else’s system.
- Recall@20 on the customer-specific golden question set
- ≥ 0.90Recall@20 on the customer-specific golden question setDesign target, measured on a labelled set built from real user questions before launch
- Answered claims without supporting cited evidence
- < 2%Answered claims without supporting cited evidenceDesign target, scored by the groundedness verifier over a held-out sample and spot-audited by hand
- End-to-end answer latency, first token to complete citation
- p95 < 3.5 sEnd-to-end answer latency, first token to complete citationDesign target for hybrid retrieval plus cross-encoder rerank plus streamed synthesis on a mid-size corpus
- Token spend avoided through caching and model tiering
- 30–60%Token spend avoided through caching and model tieringTypical range for this class of system where query traffic clusters; measured per deployment, not assumed
Stack
What this is built with
Named, current technology. Substitutions are normal — the shape of the system matters more than the vendor behind any one box.
- PostgreSQL with pgvector
- OpenSearch (BM25, metadata filtering)
- Qdrant (alternative ANN store at higher vector volumes)
- Docling and Unstructured for layout-aware parsing
- Amazon Textract for scanned and handwritten documents
- BAAI bge-m3 embeddings
- bge-reranker-v2-m3 / Cohere Rerank
- Claude via Amazon Bedrock or the Anthropic API
- vLLM for self-hosted rerank and small-model inference
- LiteLLM for provider routing and fallbacks
- Temporal for durable ingestion workflows
- Ragas and promptfoo for the evaluation harness
- Langfuse and OpenTelemetry for tracing
- Redis for exact and semantic response caching
Timeline
To a first production cut.
Four phases, run in sequence. The first one is not engineering — it is deciding precisely what the system owes, because that is what every later phase is measured against.
1–2 weeks
Corpus survey and golden set
A census of every source, format and permission model in scope, plus 100–200 real questions with labelled answers and source documents. This set is written before any retrieval code and becomes the definition of done.
3–4 weeks
Ingestion and hybrid index
Connectors, layout-aware parsing with OCR fallback, structure-driven chunking, both indexes populated with ACL tuples, and a retrieval-only harness reporting recall@k and nDCG against the golden set.
3–4 weeks
Grounded synthesis and abstention
Reranking, structured citation with span verification, the abstention policy and its thresholds, conflict surfacing, and the full answer-quality eval running in CI as a merge gate.
2–3 weeks
Hardening and production cut
Caching, model tiering and token budgets, tracing and feedback capture, load testing at expected concurrency, a documented model-upgrade runbook, and a limited rollout behind a feature flag with shadow traffic comparison.
Keep reading
Related blueprints.
Systems that share a spine with this one — the same evaluation, budget and rollback discipline, applied to a different problem.
Next step
Walk this architecture against your constraints.
A 45-minute session on the Grounded Answer Engine: which layers you already have, which ones you do not, and the decisions on this page that would go the other way for you.