Why AI pilots stall between demo and production

The demo worked. Everyone in the room agreed it was impressive. Eight months later the pilot is still a pilot, the budget line has been renamed twice, and someone has started saying the word “foundations”. This is the most common shape of AI failure in 2026, and it is almost never a model problem.

The gap between a demo and a production system is not a gap in capability. It is a gap in accounting. A demo answers one question — can this thing produce a good output at least once? A production system has to answer a different set of questions continuously, in public, while the underlying model, the data and the user population all move underneath it.

Stalled pilots tend to stall in the same six places. None of them are exotic. All six are absences rather than defects, which is exactly why they survive a steering-committee review: you cannot see a missing thing in a slide.

The measurement gap#

Start with what the demo actually measured, and what the system will be measured on once it is real. These are rarely the same quantity, and the substitution is usually invisible to everyone except the engineer who built the thing.

1. There is no eval set#

Almost every stalled pilot is being evaluated by the person who built it, reading outputs, deciding they look right. That is a vibe check on a sample the builder chose, and it has no power to detect regression. The first prompt change that improves three cases and quietly breaks nine will ship unnoticed.

A real eval set is boring and specific. Between 150 and 400 examples is usually enough to start, stratified so that roughly a third of them are known failure classes rather than representative traffic — ambiguous inputs, out-of-scope questions, documents with tables, inputs in a second language, the two customers whose data is shaped oddly. Freeze it. Version it in the repository next to the prompt.

The judging matters as much as the set. Deterministic checks — did it cite a real document id, is the JSON valid against the schema, did it stay under the token budget — should be assertions, not model calls. Reserve an LLM judge for the genuinely subjective dimension, pin the judge model to a dated snapshot, and calibrate it once against human labels so you know its agreement rate before you trust it.

// evals/run.ts — the gate between a prompt change and production traffic.
import { readFileSync } from "node:fs";
import { answer } from "../src/pipeline";
import { judge } from "./judge";

type Klass = "core" | "ambiguous" | "out_of_scope" | "multilingual" | "tabular";

type Case = {
  id: string;
  input: string;
  klass: Klass;
  // Deterministic expectations are assertions, never model calls.
  mustCite?: string[];
  mustAbstain?: boolean;
};

const SUITE: Case[] = JSON.parse(readFileSync("evals/golden.v7.json", "utf8"));
const BASELINE: Record<Klass, number> = JSON.parse(
  readFileSync("evals/baseline.v7.json", "utf8"),
);

// Per-class floors. One aggregate number hides the class that actually broke.
const FLOORS: Record<Klass, number> = {
  core: 0.92,
  ambiguous: 0.8,
  out_of_scope: 0.95, // refusing correctly is the easiest behaviour to regress
  multilingual: 0.75,
  tabular: 0.7,
};

const NOISE = 0.02; // measured by re-running the suite twice, not guessed

const mean = (xs: number[]) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : 0);

async function run() {
  const scores = new Map<Klass, number[]>();
  const record = (k: Klass, s: number) => scores.set(k, [...(scores.get(k) ?? []), s]);

  for (const c of SUITE) {
    const out = await answer(c.input);

    if (c.mustAbstain && !out.abstained) { record(c.klass, 0); continue; }
    if (c.mustCite?.some((id) => !out.citations.includes(id))) { record(c.klass, 0); continue; }

    // Judge pinned to a dated snapshot. Upgrading it is a migration, not a patch.
    const verdict = await judge({ input: c.input, output: out.text, rubric: "v7" });
    record(c.klass, verdict.score);
  }

  let failed = false;
  for (const klass of Object.keys(FLOORS) as Klass[]) {
    const got = mean(scores.get(klass) ?? []);
    const floor = FLOORS[klass];
    const was = BASELINE[klass];

    if (got < floor || got < was - NOISE) {
      console.error("FAIL", klass, "got", got.toFixed(3), "floor", floor, "baseline", was);
      failed = true;
    }
  }

  process.exit(failed ? 1 : 0);
}

run();

2. There is no cost model#

Pilots are costed per model call because that is the number the pricing page shows. Production is costed per resolved task, and the ratio between them is the thing that kills budgets. A single support resolution in an agentic system is rarely one call. It is a query rewrite, an embedding, a retrieval fan-out, a rerank, two or three tool calls, a synthesis pass, and — on maybe one attempt in eight — a retry after a schema validation failure.

Write the arithmetic down before you write the integration. A twenty-line cost model that you can argue with beats a spreadsheet nobody owns, and it belongs in the repository so that a change in fan-out shows up in a pull request.

"""cost_model.py — cost per resolved task, not cost per call.

Lives in the repo so a change in retrieval fan-out shows up in a pull request.
Run it in CI and fail the build when p95 crosses the budget.
"""

from dataclasses import dataclass, field
from statistics import quantiles

# Per million tokens. Substitute your provider's current list prices —
# these are placeholders, and they move.
PRICE = {
    "gen.in": 3.00, "gen.out": 15.00,
    "small.in": 0.80, "small.out": 4.00,
    "embed.in": 0.02, "rerank.in": 0.05,
}

CACHE_READ_DISCOUNT = 0.10  # cached prefix tokens bill at a fraction of input


@dataclass
class Turn:
    """One model turn inside a task: a rewrite, a tool call, a synthesis pass."""
    model: str
    prompt_tokens: int
    output_tokens: int = 0
    cached_prompt_tokens: int = 0


def turn_cost(t: Turn) -> float:
    fresh = max(t.prompt_tokens - t.cached_prompt_tokens, 0)
    cached = t.cached_prompt_tokens * CACHE_READ_DISCOUNT
    inp = (fresh + cached) * PRICE[t.model + ".in"] / 1_000_000
    out = t.output_tokens * PRICE.get(t.model + ".out", 0.0) / 1_000_000
    return inp + out


def task_cost(top_k: int, agent_turns: int, retry_rate: float) -> float:
    """One user task end to end, including the calls nobody counts."""
    turns = [
        Turn("small", 600, 60),                 # query rewrite
        Turn("embed", 40),                      # embed the rewritten query
        Turn("rerank", 220 * top_k),            # rerank is linear in fan-out
    ]
    turns += [
        # Static system prompt cached; retrieved chunks are not.
        Turn("gen", 1_800 + 900 * top_k, 400, cached_prompt_tokens=1_500)
        for _ in range(agent_turns)
    ]
    # A schema-validation failure replays the synthesis turn, not the whole task.
    return sum(turn_cost(t) for t in turns) * (1 + retry_rate)


if __name__ == "__main__":
    # A distribution, not a point estimate. The tail is what the budget meets.
    shapes = [(6, 2)] * 70 + [(6, 3)] * 20 + [(12, 4)] * 8 + [(20, 6)] * 2
    sample = sorted(task_cost(k, n, retry_rate=0.12) for k, n in shapes)
    p50, p95 = quantiles(sample, n=20)[9], quantiles(sample, n=20)[18]
    print(f"p50 ${p50:.4f}  p95 ${p95:.4f}  tail ratio {p95 / p50:.1f}x")

Two things fall out of this exercise immediately. The first is that retrieval fan-out, not generation, is often the dominant term — doubling top-k to fix a recall complaint can cost more than switching to a larger model. The second is that the mean is the wrong statistic. Budget against the p95 task, because the long tail is where agent loops run five turns instead of two.

Prompt caching changes this arithmetic sharply when the system prompt and the retrieved corpus are stable across a session, and it is worth designing the prompt layout around — static content first, volatile content last — before you start negotiating on model price.

3. Nobody owns quality#

Ask a stalled pilot team who decides whether output quality is acceptable this week. The honest answer is usually that quality is a shared responsibility, which is the organisational form of nobody. Shared responsibility works for things with an obvious alarm. It fails for slow degradation, and model-backed systems degrade slowly: a vendor ships a new checkpoint, the corpus drifts, a prompt is patched to fix a complaint, and nothing pages anyone.

The fix is structural, not cultural. Name one person. Give them a dashboard that shows eval scores per release, abstention rate, escalation rate and cost per resolution. Give them the authority to block a deploy. This is the same move SRE made twenty years ago with error budgets, and it works for the same reason — it converts an argument about taste into a number with an owner.

4. There is no rollback story#

Ask how to revert the system to last Tuesday’s behaviour. In a stalled pilot the answer involves someone remembering what the prompt used to say. Prompts, retrieval parameters, tool schemas, judge rubrics and model identifiers are deployment artefacts with the same blast radius as application code, and they need the same machinery.

• Pin model identifiers to dated snapshots. Never point production at a floating alias — a silent vendor upgrade is an unreviewed deploy of the most important dependency you have.
• Keep prompts in the repository, not in a database row an operations user can edit at 6pm on a Friday.
• Version the retrieval index alongside the prompt. A rollback that reverts the prompt but leaves a re-chunked index in place has not rolled anything back.
• Record the full configuration hash — model, prompt, index version, temperature, tool schema — with every response you log, so a complaint three weeks later is reproducible.
• Run the new configuration in shadow against live traffic before it serves anyone, and diff the outputs. Shadow mode is cheap relative to a public incident.

5. There are no data contracts#

The demo ran on an export. Someone pulled a CSV, removed the rows with nulls, fixed two date formats by hand, and never wrote any of that down because at the time it was not a system, it was a Tuesday afternoon. Production reads the live table, and the live table has a nullable column that became non-nullable in March, a free-text notes field containing customer phone numbers, and a nightly job that occasionally lands four hours late.

This is where most retrieval systems actually fail, and the failure is silent: the index builds, the query returns documents, the answers are merely worse. Nothing throws.

1. Declare the schema the pipeline expects, as code, and validate on ingestion rather than discovering violations at query time.
2. Set a freshness SLA per source and alert on staleness, not just on job failure — a job that succeeds on yesterday’s partition is the more dangerous outcome.
3. Test the contract in the producing team’s CI, so a schema change breaks their build rather than your answers.
4. Classify PII at ingestion and decide explicitly what is allowed into an embedding. An index is a copy of your data with none of the access controls attached to the original.
5. Track row counts and null rates per batch. A source that silently drops half its rows looks identical to a quiet week.

6. The failure mode is undefined#

The last absence is the one that does the most reputational damage. Ask what the system does when it does not know. In a stalled pilot the answer is that it answers anyway, fluently and wrongly, because nobody designed a path for the negative case.

Abstention is a feature you build, not a behaviour you request in a prompt. It needs a signal — retrieval score below a threshold, a self-consistency check across two samples, a classifier that flags out-of-scope intents, a validator that fails on unresolvable entities. It needs calibration, because a raw model confidence score is not a probability and should never be treated as one. And it needs somewhere to go.

That last point is where designs usually collapse. Escalation to a human is only a design if a human has capacity. A system that escalates fifteen per cent of traffic to a queue staffed for three per cent has not added a safety net, it has moved the failure somewhere with a worse feedback loop. Size the human loop against the abstention rate you actually measure on the eval set, not the one you hope for.

The checklist#

Run this against your own pilot this week. It is deliberately answerable in an afternoon, and the answers are usually more uncomfortable than expected.

1. Show me the eval set. How many examples, who wrote them, when was it last run, and what was the score on the last two releases?
2. What does one resolved task cost at p95, not per call — and which term dominates?
3. Who can block a release on quality, by name, and what number do they look at?
4. How do we get back to last Tuesday’s behaviour, and how long does it take?
5. Which upstream schema change would break this system silently, and whose CI would catch it?
6. What happens when the system does not know? Show me the abstention path and the queue it escalates into.

If a question has no answer, that is the work. None of these six items requires a better model, a larger budget or a new vendor. They require about three weeks of unglamorous engineering, which is precisely why they are still missing eight months in — nobody ever got applauded in a steering committee for shipping a rollback procedure.

The pilots that cross over are not the ones with the most impressive demo. They are the ones where somebody did the accounting early, while it was still cheap to be wrong.