Applied AI & Agents
Ship AI that survives contact with production traffic
Most AI projects die somewhere between the demo and the second week of real traffic. We build agentic systems, retrieval that cites its sources, and the evaluation harness that proves the thing still works after the next model upgrade.
Engagement at a glance
- Capabilities
- 05
- Deliverables
- 08 artefacts
- Stack groups
- 04
- Ways to start
- 03
Engagement options
- Evaluation & Feasibility Sprint2–3 weeks
- Production Pilot8–12 weeks
- Embedded AI EngineeringRolling quarters, minimum three months
What breaks
The failure modes we keep meeting
Cheap to design around at the start. Expensive to discover once the system is carrying real traffic.
The demo works because you wrote the demo
A prototype is tested against the twenty prompts its author thought of. Production sends the other ten thousand — truncated inputs, mixed languages, pasted spreadsheets, questions the corpus cannot answer. Without a held-out eval set built from real traffic, nobody can say whether a change made the system better or simply different. Teams end up shipping on vibes and rolling back on complaints.
Retrieval is the bottleneck, not the model
Most wrong answers are retrieval failures wearing a model’s voice. Fixed-size chunking severs tables from their headers, a single dense index misses exact identifiers that BM25 would have caught, and stale documents outrank current ones because nothing encodes recency. Then context rot sets in — stuff the window with forty passages and accuracy falls, because the relevant one is buried in the middle where attention is thinnest.
Nothing fails loudly
A non-deterministic system degrades quietly. A provider ships a new model version, tone shifts, an output schema drifts, and a downstream parser starts silently dropping fields. Meanwhile an agent loops on a failing tool and burns six figures of tokens over a weekend because no budget ceiling existed. The first signal is a customer, an auditor, or the invoice.
Capabilities
What the work actually consists of
01
Agent architecture and tool use
An agent is a control loop with a budget, not a personality. We design the loop first — what state it carries, which tools it may call, how many steps it gets, what it does when a tool returns garbage — then make the model the smallest part of the system.
- Tool contracts defined as typed schemas, with validation and idempotency keys so a retried call cannot double-charge or double-send
- Model Context Protocol servers so a tool built once is reachable from every agent and every environment
- Step, depth, wall-clock and token ceilings per run, with deterministic termination and a structured failure result rather than a silent stall
- Long-running work moved onto a durable workflow engine so an agent can resume after a deploy instead of restarting from turn one
- Multi-agent decomposition only where it earns its latency — most problems are one well-scoped agent with better tools
02
Retrieval that cites its sources
Grounding is an engineering problem with measurable stages: parse, chunk, index, retrieve, rerank, assemble. We measure each stage separately, because a system that is right 70% of the time usually has one stage that is right 70% of the time.
- Layout-aware parsing for PDFs, tables and scans, with structure preserved rather than flattened into a wall of text
- Hybrid retrieval — lexical plus dense, fused — so exact identifiers, part numbers and clause references stop disappearing
- Cross-encoder reranking over a wide shortlist, so the context window holds the best five passages instead of the nearest forty
- Citation coverage enforced at generation time: every claim traced to a retrieved span, and an abstention when it cannot be
- Freshness and permission filters applied at query time, so the index cannot leak a document the user is not entitled to read
03
Evaluation harnesses and regression suites
You cannot operate what you cannot score. We build the eval set before the feature, treat it as production code, and run it on every pull request — so the argument about whether a prompt change helped is settled by a number.
- A versioned dataset of golden cases, adversarial cases and real failures, with the reasoning for each expected answer recorded
- Deterministic checks first — schema validity, citation presence, refusal correctness, latency — before any model-graded scoring
- LLM-as-judge rubrics calibrated against human labels, with the judge’s own agreement rate measured and reported
- CI gates that fail the build on a score regression, and a shadow-traffic mode for evaluating a candidate model on live inputs without exposing it
- Pairwise comparison runs for prompt and model changes, so improvements are demonstrated rather than asserted
04
Guardrails and human-in-the-loop
The question is never whether the model will be wrong. It is what the system does when it is — and which decisions were never the model’s to make alone.
- Input and output policy screening for PII, prompt injection and disallowed content, applied at the boundary rather than inside the prompt
- Explicit abstention paths — the system says it does not know, and routes the question, instead of improvising
- Confidence and risk routing: low-stakes answers auto-deliver, high-stakes ones queue for review with the evidence attached
- Review interfaces designed for throughput, where a reviewer’s correction is captured as a labelled eval case rather than lost in a ticket
- Immutable decision logs — prompt, retrieved context, tool calls, model version, output — retained for audit and incident reconstruction
05
LLMOps, routing and cost governance
Token spend behaves like cloud spend in 2012: unmetered, unattributed and growing. We put a gateway in front of it, route each class of request to the cheapest model that passes its evals, and make cost a first-class metric.
- A single inference gateway with provider failover, retries with jitter, and per-tenant and per-feature budget caps enforced in-path
- Tiered routing — a small model for classification and extraction, a frontier model only where the evals prove it is required
- Prompt caching and context compaction designed into the prompt layout, with cache hit rate tracked as an operational metric
- OpenTelemetry traces carrying prompt, context, tool calls, tokens and cost, landing in the observability stack you already run
- Drift alerting on quality scores, refusal rate, latency percentiles and cost per resolved request — not just on uptime
Outcomes
Targets, and where each number comes from
The basis line under every figure is the point of this section. Where a number is a design standard rather than a measured delivery, it says so — we would rather be checkable than impressive.
- Inference cost reduction from tiered routing and prompt caching
- 30–60%Inference cost reduction from tiered routing and prompt cachingDesign target derived from published provider pricing — small-model tiers and cached prefix tokens are priced far below frontier input tokens. Actual saving depends on traffic mix.
- How often the regression suite scores the system
- Every mergeHow often the regression suite scores the systemDesign standard we hold ourselves to — the eval suite is a CI gate on every pull request, not a pre-release ritual.
- Grounded answers carrying a traceable citation, or an explicit abstention
- 100%Grounded answers carrying a traceable citation, or an explicit abstentionDesign target enforced by a citation-coverage check in the generation path, not a measured result from a delivered engagement.
- Discovery to a production agent behind a feature flag
- 10–15 weeksDiscovery to a production agent behind a feature flagPlanning estimate for one scoped use case — the 2–3 week evaluation sprint plus the 8–12 week production pilot below — assuming data access and a named domain reviewer are available in week one.
Stack
What we reach for, and when
Defaults, not dogma. The list below is what we would propose on a blank page; an existing estate, a procurement constraint or a team's operating experience all legitimately move it.
- Claude — Opus, Sonnet and Haiku — via the Anthropic API
- Amazon Bedrock and Google Vertex AI where inference must stay inside a VPC or a region
- OpenAI and Gemini models where a capability gap justifies a second provider
- Open-weight Llama, Qwen and Mistral served on vLLM or TensorRT-LLM
- LiteLLM or Portkey as the gateway — one interface, budgets, failover, audit
- Postgres with pgvector for teams who would rather not run a second datastore
- Qdrant or Vespa when filtered search and scale outgrow Postgres
- OpenSearch or Elasticsearch for hybrid BM25 and kNN with rank fusion
- Docling, Unstructured and layout-aware OCR for documents that fight back
- Cohere Rerank, Voyage or a bge cross-encoder over the shortlist
- Model Context Protocol servers as the reusable tool layer
- LangGraph, the Claude Agent SDK or a hand-rolled loop — whichever the problem actually needs
- Temporal for durable, resumable, long-running agent workflows
- Pydantic and Zod schemas as the enforced contract on every tool boundary
- SQS, Redis Streams or Kafka for queueing, backpressure and replay
- Braintrust, Langfuse or LangSmith for traces, datasets and scoring
- promptfoo, Ragas and DeepEval running inside CI
- OpenTelemetry GenAI semantic conventions into Grafana, Datadog or CloudWatch
- Bedrock Guardrails, Llama Guard and Microsoft Presidio for policy and PII
- Dashboards for cost per resolved request, p95 latency, refusal rate and score drift
Deliverables
What you keep
Everything below lands in your repositories and your accounts, under your licence, with the reasoning written down. There is no runtime you have to keep renting from us.
08 artefacts, handed over
- A versioned evaluation set — golden cases, adversarial cases and the failures found in staging — committed to your repository with the reasoning for each expected answer
- A regression harness wired into CI that fails the build when any score falls below its agreed floor
- The agent or retrieval service itself: containerised, instrumented, with infrastructure as code for every environment it runs in
- A retrieval evaluation report — recall@k, chunking strategies compared, reranker lift measured — with every number reproducible from the committed scripts
- A model routing and cost policy: which request class goes to which tier, with per-tenant budget ceilings enforced at the gateway
- Full request tracing — prompt, retrieved context, tool calls, model version, tokens, cost — plus the dashboards and alerts built on top of it
- A guardrail and escalation specification stating what the system refuses, what it abstains from, what reaches a human, and how that decision returns to the caller
- A runbook and handover session covering failure modes, rollback, how to add an eval case, and how to qualify the next model release
Engagements
Three honest ways to start
Sized so the first one can end. Each option is designed to produce something usable even if we never work together again.
Evaluation & Feasibility Sprint
2–3 weeks
You have a prototype, or a strong opinion, and need to know whether it holds up before committing a budget to it.
Includes
- Use-case triage against a cost, risk and measurability grid
- A first evaluation set built from your real inputs, not invented ones
- Baseline scores for the current approach, with the failure modes named
- A retrieval and architecture recommendation with the tradeoffs written down
- A costed delivery plan, or an honest recommendation not to build it
Production Pilot
8–12 weeks
One use case, taken from nothing to real users behind a feature flag, with the operational scaffolding that makes the second one faster.
Includes
- The agent or grounded retrieval service, built, evaluated and deployed
- CI-gated eval suite, guardrails and human review path
- Inference gateway with routing, caching and budget enforcement
- Observability, alerting and the cost model for steady-state running
- Runbook, handover and a working session with the team who will operate it
Embedded AI Engineering
Rolling quarters, minimum three months
You have several AI surfaces in flight and need standards, reviews and platform work alongside delivery rather than after it.
Includes
- A shared eval, tracing and gateway platform across teams and use cases
- Model release qualification — every provider upgrade scored before it reaches production
- Architecture review and pairing with your engineers on their own AI work
- Cost governance reporting per team, feature and tenant
- Quarterly reassessment of model, retrieval and vendor choices against the evidence
Questions
The awkward ones
The questions that decide whether this is worth starting — answered the way we would answer them on a call.
It is usually the smaller part of the work. The prototype proves the idea is possible; the remaining work is proving it is reliable — eval sets, retrieval quality, guardrails, cost ceilings, tracing, rollback. The gap is rarely model capability. It is the absence of any mechanism to detect that today’s version is worse than yesterday’s.
Adjacent work
These are the practices this one leans on and the reference architectures that show it assembled.
Related solutions
Reference architectures
Next step
Bring us the problem you keep deferring.
A 45-minute working session on your Applied AI & Agents work. We will tell you what we would build, what we would not build, and roughly what it costs. No deck.