Most AI pilots stall because they are built for the demo and not for the business. They run on a tidy sample of data, nobody has written down what a right answer looks like, and nobody owns them once the excitement fades. Production asks for the opposite: the real data in all its mess, a way to prove the AI is right, a route into the systems people already use, and a named person who answers for it. A pilot that skipped those has nowhere to go but a slide.
I’ve been on both sides of this. At a telco I led the machine-learning work on a real-time call-steering system that went into production for millions of customers. Years later I built a market-data platform the other way round: five clever model engines on top, and no discipline at all for the data feeding them. The first worked because the plumbing came first. The second never earned its keep, so I threw it away and started again. What follows is what the difference was.
What does a pilot actually prove?
Usually, that a model can do the task on the examples someone chose. That is worth knowing, and it is the smallest thing you need to know. It says nothing about the cases nobody picked, about what happens when the data arrives late or wrong, or about whether anyone will change how they work because of it. A pilot that answers only the first question is a proof of concept, and it should be called one, so that nobody plans a launch on it.
Why does the data stop it?
Because the pilot ran on a copy someone cleaned by hand, and production runs on the real thing. The real thing has duplicates, gaps, fields that changed meaning halfway through a year, and identifiers that get reused. In that first market-data platform, the cleaned data layer ended up with more than ten thousand cases of the same fact recorded twice, and some fifteen hundred where the two copies disagreed. The models sat on top of that. None of them established anything, and I can’t tell you whether that was the models or the data, which is exactly the problem.
Data is usually the slowest part to put right, so it is the part a pilot is most tempted to skip. Skipping it is a loan, and production calls it in.
How do you know the AI is right?
You decide what right means before you build anything, and you write it down as cases the system must pass. This is usually called an evaluation set. Mine are labelled pairs: real examples, each with the answer a careful person would give and a note on why. On the second version of that platform, I built the harness that grades the record matching before I built the matcher. The first version’s plan had asked for exactly that and never got round to it. Its matcher then proposed merging things that were plainly different, two share classes of one company among them, and four bonds filed as shares.
An evaluation set does two jobs. It tells you whether the pilot is good enough, and it tells you, every time someone changes a prompt or swaps a model, whether it still is. Without one, every change is a matter of opinion, and the loudest opinion in the room wins.
A pilot with no written definition of a right answer is a demo with a budget.
What else goes missing between pilot and production?
The parts nobody puts on the slide. These are the ones I check first:
- An owner. One person who answers for it after the project team has moved on, and who decides when it is wrong.
- A way in. The pilot lived in its own tool. Production means the systems people already open every morning, with their logins and their permissions.
- A record. What it saw, what it said, and who acted on it. In regulated work this is not optional, and it is far cheaper to build in than to bolt on.
- A safe way to be wrong. A person who reviews before anything irreversible happens, or a rehearsal mode that shows what would change before it does.
- A running cost. The bill for the model, the data and the person who keeps it alive, set against what it saves.
None of these is hard on its own. Together they are most of the work, which is why a pilot that took a few weeks can take many months to ship. Projects die in that gap, because the budget was set by the weeks.
Is it the model’s fault?
Rarely, in my experience. Models are good now, and they get better every few months without any help from you. What they can’t do is make up for data nobody trusts or a question nobody has defined. When a pilot disappoints, I look at the inputs and the definition of success before I look at the model. Swapping models is the expensive way to find out the problem was upstream.
How do you run a pilot that can actually ship?
Build it as the first slice of the production system, rather than as an experiment beside it. In practice that means:
- Start from the decision it will change, and from the person who makes that decision today.
- Use real data from day one, even a small slice of it, arriving by the same route production will use.
- Write the evaluation set before the first prompt, and keep it in the repository beside the code.
- Give it a rehearsal mode, so it proposes and a person applies, until the record shows it can be trusted with more.
- Agree up front what result would make you stop, and then honour it.
This is slower for the first fortnight and much faster after it, because you never have to rebuild the pilot into the product. It already is the product, only small.
The short version
AI pilots mostly fail in the foundations. Get the data honest, define a right answer before you build, and give the thing an owner and a way into real work. Then a pilot becomes the first floor of something, instead of a model on a table. If yours has stalled, the crack is usually visible from the ground, and finding it is the work I like best.