The pattern is so consistent it's almost boring. A company runs an AI pilot. The demo is impressive. Everyone nods. Six months later the pilot is quietly shelved, the budget line disappears, and the organisation concludes that "AI isn't ready for our industry." The model was rarely the problem. The pilot design was.
Industry surveys keep putting the share of AI pilots that never reach production somewhere between seventy and ninety percent, and our own experience inheriting stalled pilots says the surveys are roughly right. What the surveys don't say is that the deaths are predictable. The same three design flaws account for most of the graveyard — and all three are visible in the pilot's first week, long before any model has had the chance to underperform.
The three ways pilots die
Death by missing metric. The pilot was chartered to "explore AI capabilities" — which means nobody defined the number it had to move. Without a metric, there is no finish line; without a finish line, budget review kills it by default. A pilot should be an experiment with a hypothesis: "this model can cut first-pass review time by half at equal accuracy." Pass or fail, you learn something budgetable.
The tell is in the charter's language. "Explore," "evaluate the potential of," "build capability in" — these are phrases that cannot fail, and anything that cannot fail cannot succeed either. Compare: "reduce median exception handling time from 40 minutes to under 15, measured on live volume, by end of Q2." That sentence can lose. Which is exactly what makes it fundable when it wins.
Death by demo data. The pilot ran on clean, curated examples. Production runs on scanned faxes, forwarded email chains and the one supplier who still sends TIFFs. The gap between demo accuracy and production accuracy is where trust dies — and once operators stop trusting the tool, no accuracy improvement wins them back. The fix is unglamorous: build the evaluation set from your ugliest real cases first.
There's a deeper reason this kills pilots: the demo-data gap doesn't show up as a single failure, it shows up as an erosion. The tool is right on the easy cases — which the team could handle anyway — and wrong on the hard ones, which is where they actually wanted help. Operators do the arithmetic quickly. Within a month the tool has a nickname, and the nickname is not affectionate.
Death by missing owner. The pilot belonged to an innovation team; the workflow belonged to operations. When the pilot ended, no one whose bonus depends on the process was accountable for adopting it. Pilots survive when the process owner — not the sponsor — asks for them.
Two more failure modes deserve honourable mention. Death by wrapper: the pilot lives in a separate tab, a separate login, a separate window — and every context switch it demands is a small toll on adoption. If the model's output doesn't appear inside the tool where the decision is made, it will be consulted for a week and then forgotten. And death by late governance: security, legal and compliance hear about the pilot in month four, at which point their only available move is to stop it. Bring them the data-flow diagram in week one and they become collaborators; surprise them and they become, correctly, a wall.
A pilot is not a small demo. It's a small production system — with a metric, real data and an owner.
Choosing a workflow that can carry a pilot
Half of pilot success is decided before any vendor is called, in the choice of workflow. The good candidates share four properties. Volume: the task happens hundreds of times a month, so wins are measurable and training signal accumulates. Tolerance for review: a human can cheaply check the output — summarise, classify, extract, draft — rather than tasks where verification costs as much as doing the work. Existing data: the historical cases needed for an evaluation set already sit in a system somewhere, not in the heads of two veterans. Pain: the people doing the task today hate the worst part of it, because their motivation is the adoption engine no rollout plan can replace.
The bad candidates are just as recognisable. Anything safety- or compliance-critical as a first project — not because AI can't help there, but because the first project sets the organisation's trust budget, and a visible early failure in a high-stakes flow spends it all. Anything with fewer than a hundred cases a month, where the statistics will never separate signal from luck. And the CEO's favourite demo from a conference, which has the sponsorship but rarely the volume, the data or the owner. Pick the boring, high-volume, hated workflow every time.
The feasibility sprint, step by step
The two-week sprint that opens every AI engagement we run has a fixed shape, and each step exists because skipping it has burned someone. Days one and two: collect the evaluation set — one to two hundred real cases, sampled from live volume, with the operators pointing out which ones are hard and why. Not the cases the sponsor forwards; the cases the team dreads. Days three and four: baseline the humans. How long does the task take today, what does it cost, what's the current error rate? Without this number the pilot cannot prove anything later.
The middle week: run the candidate models against the evaluation set and build the error taxonomy — not just "87% accurate" but which 13% fails, and whether the failures cluster somewhere cheap (rare document types you can route to humans) or somewhere fatal (the highest-value cases). The final days: price the unit economics — cost per case at production volume, including the human review the confidence routing will trigger — and write the go/no-go memo. Roughly a third of our sprints end in "no-go," and those memos have saved clients more money than some of the systems we've shipped.
Three that made it
Contract review at a law firm. The charter named one number: first-pass review time. The evaluation set was built from the firm's own past contracts, including the messy ones, and scored against the partners' own past markups. Lawyers saw the model's confidence on every clause and could reject suggestions in one click — rejections fed back into evaluation. Adoption wasn't mandated; it happened because the tool made the annoying part of the job shorter.
Exception triage at a retailer. Instead of aiming the model at the whole order flow, it was aimed only at exceptions — the 4% of orders that ate 60% of the team's time. Confidence-based routing meant the model handled the routine exceptions and escalated the strange ones. Nobody's job changed except the worst hour of it.
Demand forecasting embedded in purchasing. The previous "AI dashboard" had been ignored for a year. The rebuilt version put the forecast inside the purchasing tool, at the moment of the order decision, with the model's error range shown honestly. Buyers overrode it freely — and the override rate itself became the adoption metric. It fell month over month.
Notice what the three survivors share. None of them aimed AI at an entire job — each aimed it at the worst slice of one. All three showed their uncertainty instead of hiding it, which is what let the humans calibrate trust instead of losing it. And in all three, the metric belonged to the people doing the work, not to a slide. The pattern is almost embarrassing in its simplicity; it's just rarely followed, because "transform the department" makes a better kickoff meeting than "fix the worst hour of Tuesday."
Confidence routing, the pattern that makes AI deployable
If one engineering pattern separates shipped systems from shelved ones, it's this. Every model output carries a confidence signal; the system routes on it. High confidence: straight through, logged. Medium: into a human review queue, pre-filled, one keystroke to approve. Low, or a case type the eval set never saw: escalate entirely, and flag the gap. The thresholds are business decisions, not technical ones — they encode how expensive a mistake is, and they differ between a marketing draft and a payment instruction.
Routing does three jobs at once. It caps the blast radius of model errors, which is what makes the system defensible in front of a risk committee. It generates labelled training data as a by-product of normal work — every human approval or correction sharpens the next evaluation round. And it gives adoption a gradient: the team watches the machine earn trust case by case, instead of being asked to grant it upfront. Systems introduced this way get more autonomous over time, with evidence at each step. Systems introduced as "fully automated" get switched off at the first visible mistake.
What a pilot should cost
A pilot that cannot fail cheaply will not be allowed to fail at all — it will be declared a success and quietly unused, which is worse. The cost structure should match the decision structure: a feasibility sprint priced in weeks, a production build priced in milestones that each ship something reviewable, and running costs modelled per case before go-live, not discovered on the first invoice. Model usage is a variable cost, and workflows with heavy documents or long contexts can surprise you at production volume; the sprint's unit-economics step exists precisely so the surprise happens on a spreadsheet.
Beware the opposite failure too: the pilot that is too small to prove anything. A weekend prototype on ten hand-picked cases produces exactly one artefact — a misleading demo. If the workflow matters enough to pilot, it matters enough to baseline properly and evaluate on real volume. The minimum honest pilot is smaller than most companies fear and larger than a hackathon: weeks, not days; hundreds of cases, not ten.
What to measure after go-live
The pilot metric answers "did it work in month one." Production needs a dashboard that answers "is it still working in month twelve," and the interesting numbers are behavioural. Override rate: falling is health, rising is drift — in the data, the model, or the team's trust. Escalation mix: if the human queue is growing, either volume shifted or the model stopped covering it. Unit cost per case, because model pricing and usage patterns both move. And accuracy on a rotating slice of live cases re-scored against the eval harness, because the world your model was measured on last year is not the world it runs in now.
This is also the honest answer to "when is the AI project finished": never entirely — but the ongoing cost is small, budgetable and boring, exactly like maintenance on any other production system. What's not acceptable is the alternative most failed deployments chose: ship it, stop measuring, and find out from an angry customer that the model has been quietly wrong since March.
Give the numbers a ritual, or they won't survive the quarter. A monthly thirty-minute review — process owner, one engineer, the dashboard — is enough: what moved, what drifted, which escalations were new, and one improvement picked for the coming month. The ritual matters more than the tooling, because a dashboard nobody is scheduled to look at is just a screenshot with hosting costs. And write down what each review decided; the third time a model is swapped or a threshold moved, that log is the difference between an institution learning and a team guessing twice.
The checklist
One metric, named in the charter, that can visibly fail. An evaluation set built from your ugliest real cases, not your prettiest. A process owner who wants it, not a sponsor who funds it. Output delivered inside the tool where the decision is made, never in one more tab. Security and compliance in the room in week one. Confidence routing to humans from day one, with thresholds set by the cost of being wrong. A feasibility sprint before the build, so the go/no-go decision costs weeks, not quarters. And a post-launch dashboard someone actually owns. That's the whole difference between a pilot and a press release.