Intelligent Automation & Applied AI
Document Intelligence Pipeline
A high-volume extraction pipeline that turns invoices, claims, KYC packs and trade documents into validated structured records, attaches a calibrated confidence to every field, and sends only the fields that genuinely need a person into an exception queue sized to the real error rate.
At a glance
- Type
- Reference architecture
- Domain
- Intelligent Automation & Applied AI
- Architecture
- 6 layers
- First production cut
- 10–13 weeks, phases in sequence
A reference architecture. Figures on this page are design targets, not measured client results.
The challenge
An operations team receives thousands of documents a day in every format a counterparty can produce — a clean PEPPOL e-invoice, a scan of a fax, a photo taken on a warehouse floor at dusk. What the team actually wants is not extraction. It wants a record it can post without a human touching it, plus a small, honest pile of the ones that do need looking at. The hard problem is not reading the document — it is knowing, field by field, whether the read can be trusted.
Failure mode
The obvious approach, and why it breaks.
Almost every team reaches the same first design, and it is not a bad instinct — it is the shortest route to something that demos. Here is that design, and the point at which real volume takes it apart.
Discarded
The obvious approach
Send each page to a vision model with a prompt asking for a JSON object of the required fields, parse the reply, write it to the ERP. Add a retry when the JSON fails to parse, and call the accuracy question answered because the demo set came back clean.
Why it breaks
What production does to it
The output has no per-field confidence, so only two operating policies exist and both are bad: review every document, which deletes the saving, or review none, which lets errors reach the ledger silently. The model happily returns a syntactically perfect object with a hallucinated invoice number where the source was a smudge, and nothing downstream can tell that field apart from one it read cleanly. Line-item tables get truncated at the page break. Totals that do not add up are posted anyway, because arithmetic was never checked. No field carries a bounding box, so a reviewer who does get the document has to re-read all six pages to verify one number. Cost scales with full-page image tokens on every document, including the large share that are machine-generated PDFs with a perfect text layer. And when a large vendor changes its template, accuracy on that stream falls quietly — the first signal is a reconciliation break a quarter later.
Architecture
How the system is put together.
6 layers, from the edge where work arrives to the operations that keep it honest. Each one names the components it owns and why it exists as a separate concern.
Layer 01
Intake, classification & normalisation
A meaningful share of any real corpus never needs a model at all. Routing structured e-invoices and clean text PDFs away from the vision path before the first token is spent is the single largest cost lever in the system, and splitting batches correctly prevents a class of error no downstream layer can repair.
Components
- Channel connectors: SFTP drop, O365/IMAP mailbox, S3 event, ERP outbox, upload API
- Content-hash and perceptual-hash deduplication to catch re-sends and re-scans
- Structured-format fast path: PEPPOL UBL, Factur-X, EDI 810/850 parsed, never modelled
- Page normalisation — deskew, despeckle, orientation and DPI correction
- Document classification and page-splitting for stapled multi-document batches
- Quarantine lane for encrypted, zero-page, malware-flagged or oversized files
Layer 02
Layout parsing & OCR tiering
Everything above this layer is a function of what it produces, so nothing is allowed to collapse into a flat string. Bounding boxes and per-token confidence survive all the way to the reviewer, because a field without a coordinate cannot be verified quickly and a field without a source confidence cannot be calibrated at all.
Components
- Native text and geometry extraction (PyMuPDF, pdfplumber) when a real text layer exists
- Layout detection into blocks, tables and key–value regions (Docling, DocLayout-YOLO)
- Self-hosted OCR tier for scans (PaddleOCR, Surya) with per-token confidence
- Managed OCR tier (Amazon Textract, Azure AI Document Intelligence) for hard pages only
- Table structure recovery that preserves cell coordinates and spanning headers
- Quality gate on OCR confidence, blank-page and glyph-garbage detection, with re-tiering
Layer 03
Schema-constrained extraction
The schema is the contract, versioned alongside the code that consumes it. Constrained decoding removes an entire class of parse failures, and evidence binding turns every value into something a human — or an auditor two years later — can check against the pixels it came from.
Components
- Versioned Pydantic schema per document class as the single source of truth
- Constrained decoding (XGrammar, Outlines) so the output is valid JSON by construction
- Explicit absence: every field can return not_present with a reason, never a guess
- Evidence binding — each field carries page, bounding box and source token span
- Template fingerprinting with few-shot exemplar retrieval for known senders
- Deterministic anchor and regex extractors where a sender’s layout is genuinely stable
Layer 04
Confidence, validation & decisioning
This layer decides how large the human queue is, which makes it the commercial core of the system. A raw model score is not a probability; calibrating it against labelled outcomes is what allows a sentence like “auto-post everything above this threshold and expect roughly this error rate” to be true rather than hopeful.
Components
- Field-level confidence from token logprobs, OCR confidence and two-pass self-consistency
- Per-field, per-class calibration by temperature scaling or isotonic regression on held-out labels
- Conformal thresholds set to a target field error rate, refitted on a rolling window
- Deterministic checks: GSTIN and IBAN checksums, Luhn, date sanity, currency and locale parsing
- Cross-field arithmetic — line items to subtotal, tax to declared rate, subtotal to payable
- Reference matching: three-way match to PO and goods receipt, master-data lookup for vendor or member
Layer 05
Exception queue & human review
The queue is designed as a capacity system, not an inbox. Reason codes matter more than a single score because a failed checksum and an unseen template need different people and different fixes, and measuring the reviewers is what stops a rubber-stamping culture forming under load.
Components
- Routing by reason code — low confidence, failed check, unmatched vendor, unseen template, high value
- Field-anchored review console: source crop beside the value, keyboard-first, no whole-document re-keying
- Work-conserving priority by monetary exposure and payment or regulatory due date
- Segregation of duties and four-eyes review above configured value thresholds
- Every correction persisted as a labelled example with its original evidence and model version
- Blind double-review on a sampled fraction to measure reviewer agreement, not just model accuracy
Layer 06
Learning loop, cost & audit
Without the random audit sample the loop only ever learns from documents a human already saw, and the system becomes confidently blind exactly where it auto-posts. The audit record exists so a disputed entry can be reconstructed months later instead of argued about from memory.
Components
- Correction store (Argilla, Label Studio) feeding the next calibration and fine-tune
- Random audit sample of auto-posted documents, labelled regardless of confidence
- Per-field, per-vendor drift reports with alerting on template change
- Shadow evaluation of candidate models on replayed traffic before any promotion
- Immutable audit record: input hash, model and schema versions, raw output, thresholds, reviewer
- Cost and throughput telemetry per document class, per page and per tier
Engineering decisions
Six decisions, and what each one costs.
A decision without a stated cost is marketing. These are the choices that shape the system, the reasoning behind them, and what is given up in exchange.
- 01
Tier the reading path — native text layer first, self-hosted OCR next, managed OCR only for pages that fail a quality gate.
Why
Cost and latency per document are dominated by how the pixels are read. Most corpora contain a large fraction of machine-generated PDFs whose text layer is exact and free, and sending those through a vision model is pure waste. The tiers mean spend lands on the pages that are genuinely hard.
What it costs
Three code paths to test and keep behaviourally consistent, and a routing decision that can itself be wrong. Some PDFs carry a text layer that disagrees with the rendered glyphs, so a sampled render-and-compare check is needed to catch them — a single always-OCR path would be slower and dearer but far simpler to reason about.
- 02
Constrain decoding against a versioned schema, and make absence an explicit, first-class value.
Why
Prompt-and-parse fails in two directions: malformed JSON, and well-formed JSON containing invented values. A grammar removes the first entirely. Forcing every field to be either a value with evidence or an explicit not_present with a reason removes the incentive to fill a blank convincingly.
What it costs
Grammar constraints still push a model towards emitting something valid, so schema design has to leave a graceful exit or the constraint becomes a hallucination generator. Grammar compilation adds latency on first use, narrows the set of usable providers, and every schema change is now a versioned migration with a backfill question attached.
- 03
Calibrate confidence per field and per document class, with conformal thresholds targeting a stated error rate — not one global score cutoff.
Why
Raw model scores are not probabilities, and they are wildly different across fields: a printed invoice number and a handwritten claim date do not deserve the same threshold. Calibrating against labelled outcomes lets the business choose an error rate and have the review volume fall out of it, which is the conversation finance actually wants to have.
What it costs
It requires a labelled held-out set per class and continuous relabelling to keep it alive. The statistical guarantee holds only while new documents resemble the calibration set, so a new vendor template or a replaced scanner quietly invalidates it — which forces drift detection and a deliberately conservative cold start for anything unseen. It is far more machinery than a slider anyone can explain in a meeting.
- 04
Gate on deterministic validation and cross-field checks before model confidence is trusted at all.
Why
Arithmetic, checksums and a three-way match are cheap, exact and explainable. A total that does not equal the sum of its lines is wrong whatever the model believed, and a vendor that matches no master record is an exception no confidence score should be allowed to overrule.
What it costs
Rules are a permanent maintenance surface — every new jurisdiction, tax regime and counterparty format adds more of them, and they rot silently. Over-strict rules are worse than none: they push clean documents into the queue, inflate volume, and train reviewers to approve without reading, which destroys the value of the queue itself.
- 05
Fine-tune a small layout-aware model for the high-volume head; keep a frontier vision model for the long tail.
Why
A handful of document classes usually carry most of the volume, and on those a small fine-tuned model served locally is an order of magnitude cheaper per page and materially faster. The tail is where a frontier model earns its cost — one-off formats, unusual languages, bad photographs.
What it costs
Fine-tuning needs thousands of labelled pages before it is worth anything, plus training, evaluation and rollback machinery that is real engineering. Two model families now have to be monitored, and the small model degrades ungracefully when a template shifts — it returns confident nonsense on a layout the frontier model would have handled. Routing between them becomes another thing that can be wrong.
- 06
Train the learning loop on reviewer corrections plus a mandatory random audit sample, never on corrections alone.
Why
Corrections are drawn entirely from documents the system already doubted. Learning only from them bakes in selection bias and leaves the auto-posted population unmeasured, which is precisely the population where an undetected error is expensive. A small random sample labelled regardless of confidence is the only honest estimate of the live error rate.
What it costs
It spends scarce reviewer capacity on documents that are almost always correct, which is a genuinely unpopular line item and the first thing cut when the queue backs up. Holding it requires committing the capacity in advance and defending it — the sample must also be large enough to detect the error rates being claimed, which at high accuracy is not a small number.
Design targets
The operating point we build toward.
Targets, with the basis for each one written underneath it. They describe what this architecture is designed to hold — not a result measured on somebody else’s system.
- Field precision on auto-posted documents
- ≥ 99.5%Field precision on auto-posted documentsDesign target. It is enforced by the calibrated threshold rather than promised by a model — the system reviews more documents, not fewer, until the held-out set supports it.
- Straight-through rate on mature document classes
- 60–85%Straight-through rate on mature document classesDesign target band for this class of system. The realised figure is a property of the corpus: clean structured invoices sit near the top, scanned claims and handwriting near the bottom.
- Intake to queue-ready for a ten-page document
- < 60s p95Intake to queue-ready for a ten-page documentDesign target for the standard path at steady-state load. Bulk backfills run on a separate lane with their own budget so they cannot starve live traffic.
- Blended model cost per document
- $0.02–0.10Blended model cost per documentEngineering estimate modelled from 2026 published list prices for OCR and vision inference at page counts typical of these classes, under the tiered reading path. Not a quoted price.
Stack
What this is built with
Named, current technology. Substitutions are normal — the shape of the system matters more than the vendor behind any one box.
- Docling for layout-aware document parsing
- PyMuPDF and pdfplumber for native text and page geometry
- DocLayout-YOLO for page region and table detection
- PaddleOCR and Surya for the self-hosted OCR tier
- Amazon Textract or Azure AI Document Intelligence for the hard-page tier
- Claude via Amazon Bedrock for long-tail vision extraction
- Qwen2.5-VL fine-tuned and served on vLLM for high-volume classes
- XGrammar or Outlines for schema-constrained decoding
- Pydantic v2 as the versioned schema source of truth
- Temporal for durable per-document workflows and retries
- PostgreSQL with pgvector for records, template fingerprints and queue state
- S3 with Object Lock for immutable page images and audit records
- Argilla for the labelled correction and audit-sample store
- OpenTelemetry, Langfuse and Grafana for tracing, cost and drift dashboards
Timeline
To a first production cut.
Four phases, run in sequence. The first one is not engineering — it is deciding precisely what the system owes, because that is what every later phase is measured against.
2 weeks
Document census and ground truth
A census of every channel, class, sender and language in scope with real volume weights, plus a labelled set of several hundred documents with field-level ground truth. This set defines done and is written before any extraction code.
3–4 weeks
Reading path and extraction harness
Intake, deduplication, classification, the tiered reading path and schema-constrained extraction with evidence binding, measured offline against the labelled set as field-level precision and recall per class — no ERP writes yet.
3–4 weeks
Calibration, validation and the queue
Per-field calibration and conformal thresholds, the deterministic and cross-field validation suite, reason-code routing, and the field-anchored review console running in parallel with the existing manual process for comparison.
2–3 weeks
Production cut and learning loop
Auto-posting enabled on the classes that clear their thresholds, random audit sampling in place, drift alerting per vendor template, cost telemetry, shadow evaluation for model upgrades, and a documented rollback to full review.
Keep reading
Related blueprints.
Systems that share a spine with this one — the same evaluation, budget and rollback discipline, applied to a different problem.
Next step
Walk this architecture against your constraints.
A 45-minute session on the Document Intelligence Pipeline: which layers you already have, which ones you do not, and the decisions on this page that would go the other way for you.