Summary
Most AI pilots that stall never get killed. They sit there, technically successful, going nowhere, while everyone quietly agrees not to talk about why it never became a real workflow. Deloitte found the pattern is nearly universal. Only 25 percent of surveyed organizations had moved at least 40 percent of their AI experiments into production. Even so, 34 percent said they were pursuing deep transformation with AI. McKinsey found workflow redesign had the strongest relationship with reported earnings impact among 25 attributes it tested. Only 21 percent of organizations using generative AI had redesigned even one workflow. The model usually works fine. It’s the organization around it that hasn’t finished the job.
An artificial intelligence (AI) pilot should become an operating workflow only after the surrounding business system can support it. Technical performance is one piece of that evidence. Production also requires repeatable demand, accountable owners, dependable inputs, working controls, and measurable business value. A pilot can succeed because its creators select clean examples, watch every result, and repair problems before anyone else sees them. Daily operations bring normal volume, weak inputs, competing priorities, and employees who never built the thing.
One: the demand is real, not manufactured
An operating workflow needs a steady reason to exist. Pilot teams sometimes invent scenarios or recruit friendly users to test an idea. That approach supports early learning, but it proves nothing about whether normal demand will sustain the workflow after launch.
Pull several weeks of actual requests, cases, or transactions, and answer these with real data instead of impressions:
- How many cases arrive in a normal week or month.
- Whether demand spikes seasonally in a predictable way.
- Who needs the result, and what decision follows it.
- What the work currently costs in delay or risk when it doesn’t happen.
A queue or transaction history is stronger evidence than stated interest, since people often support an idea before they understand how it changes their daily work. A lower-volume workflow can still deserve production when each case carries substantial revenue, risk, or customer consequence.
Two: you can draw a hard line around where it starts and stops
A pilot usually tests one capability. An operating workflow has to connect that capability to everything that happens before and after it. Map the work from the original request through the final business action, including every handoff, approval, and waiting period.
That exercise usually reveals hidden work the pilot quietly excluded: someone cleaning a source file, interpreting an unclear request, or reformatting output by hand. Those activities still consume capacity after the AI step succeeds, and they belong in the production design and the value calculation. The workflow also needs a defined edge. A system built to draft routine customer responses should not quietly start handling legal disputes.
The real test is simple. Can an ordinary employee, not the person who built it, describe the complete workflow, including exactly where it starts, ends, and hands off responsibility?
Three: somebody with real authority owns it
An operating workflow needs one accountable business owner with real authority over its purpose, performance, and continued use. Technology ownership alone doesn’t cover this. The owner has to understand the work well enough to approve rules, resolve disagreements, and judge whether results still hold up.
Several roles can support that owner:
- A process owner for daily execution.
- A data owner for information quality.
- A technical owner for connections and system access.
- A review owner for human validation.
- A risk owner for security and compliance concerns.
One person can hold several roles in a smaller company, but every responsibility still needs a name and available time attached to it. MIT Sloan’s CISR research identifies strategy, systems, synchronization, and stewardship as the requirements for moving AI past early pilots. None of that happens through informal enthusiasm from one employee who might change jobs next quarter. Production becomes realistic when the owner can stop the workflow, change its rules, and answer for its results.
Four: the data holds up when it isn’t hand-picked
Clean pilot data hides missing fields, conflicting records, and inconsistent naming, problems that only become visible after broader use. Production testing has to include the inputs employees receive, not the ones the pilot team curated.
Test against three groups:
- Strong inputs that are complete and current.
- Typical inputs that reflect what employees usually receive.
- Weak inputs with the gaps and formatting problems that show up in real work.
The workflow needs a defined response for each group, whether that means proceeding, requesting more information, or routing the case for review. It also needs an approved rule for which source wins when two records conflict, since a model should never make that call on its own. The real sign this is ready: the workflow recognizes inadequate input instead of producing a confident answer from incomplete material.
Five: the errors are sorted by consequence, not averaged into one number
An overall accuracy percentage hides more than it reveals. A minor formatting error and an invented customer commitment cannot share one blended score. Build an error classification tied to the workflow’s actual purpose:
- Critical errors that could cause legal, safety, or serious customer harm.
- Major errors that produce a wrong decision.
- Moderate errors contained within the team.
- Minor errors that affect presentation only.
Set an acceptable limit for each class separately, since two percent might be fine for internal tags and unacceptable for a regulated statement. OpenAI recommends contextual evaluations that define success within the specific workflow and operating environment, which is what exposes high-severity errors before wider release. NIST recommends testing under conditions similar to real deployment, with continued evaluation after any model, prompt, or source change. Leaders need to understand the remaining error profile and formally accept it. Hope is not a documented tolerance decision.
Six: review runs on rules, not on whoever happens to be free
Many pilots rely on the same experts who built the system reviewing every output personally, an arrangement that rarely survives real volume. The team needs a review model based on risk, uncertainty, and reversibility:
- Full review before every output.
- Conditional review triggered by defined risk conditions.
- Sampled review for quality monitoring.
- Exception-only review, with routine cases running automatically.
- Trend-level review of performance rather than individual cases.
Whichever model applies, it needs a named reviewer, a clear inspection standard, a response time, and real authority to correct, reject, or stop a result. NIST recommends measuring oversight, overrides, and escalations directly rather than assuming review is working because a person is technically in the loop. Review capacity belongs in the operating math too. A workflow that saves 20 employee hours and creates 25 review hours has relocated the work, not removed it.
Seven: the exceptions have somewhere to go
Every operating workflow eventually meets a case outside its normal path. Review the pilot’s history for anything that needed manual intervention or unusual judgment, and interview users about problems the test never surfaced. Build an exception register naming the condition, its expected frequency, and its required response.
Each exception needs a real destination: a subject expert, a service queue, or a request for more information. The workflow should also distinguish a known exception from a system failure. An unusual customer request needs expert review. An expired connection needs technical support instead. Unknown exceptions need a safe default: pause or escalate, never guess. A production workflow needs to manage the expected variation without silent failure or improvised decisions. It doesn’t need to eliminate every exception outright.
Eight: the systems underneath it can take the load
An AI capability usually depends on databases, document systems, customer platforms, and automation services underneath it. A strong model result provides little value if the output never reaches the next system cleanly. Document every connection, permission, and field mapping, including vendor services sitting behind the visible product.
Test real failure conditions directly: expired credentials, duplicate events, an unavailable system, and a partial failure that needs to be recorded rather than silently dropped. Security and privacy review should cover data access, retention, deletion, and model training terms, with the workflow receiving only the access its approved purpose requires. NIST recommends ongoing monitoring specifically because performance can shift after deployment. Sources, models, or business conditions can change underneath a workflow that looked stable during testing. What matters is whether the complete system can fail visibly, recover predictably, and protect information the whole way through.
Nine: people who didn’t build it can run it
Pilot creators understand why the system behaves the way it does. Everyone else only sees the interface and the output. Production testing has to include the people who will request, review, or receive the work. Their experience is what exposes unclear instructions and hidden dependencies.
Users need to know when the workflow applies and what information they must provide. They also need to know how to interpret the result and how to report a problem or fall back to the manual process. Training should use realistic, messy cases rather than a polished demo. Observe actual use directly, since attendance at a training session proves nothing about whether someone can do the work correctly. MIT Sloan’s research emphasizes coordinating people, roles, and systems together, redesigning the work around the technology rather than distributing access and hoping adoption follows. The final test is whether typical users can run it safely without daily help from the original project team.
Ten: the numbers justify keeping it running
A production decision needs evidence that the workflow improves a business result after every operating cost gets counted. Time saved is a useful signal and a poor complete story. Establish a baseline from the current process, covering volume, cycle time, correction rates, and downstream outcomes, before the pilot ever expands.
Track operating health directly:
- Completion rate and exception rate.
- Correction rate and review time.
- Failure rate and unit cost.
- Real adoption by the people the workflow was built for.
The business result has to exceed the full operating burden, including monitoring, review, support, and continued maintenance, not only the license fee. McKinsey’s research on workflow redesign supports measuring the complete process rather than counting isolated model activity by itself. Leaders should be able to say what improved, how much, what it required, and which result would trigger revision or retirement.
What the review should decide
A readiness review only has three honest outcomes. Controlled production fits a workflow with repeatable demand, clear ownership, dependable inputs, accepted error limits, working controls, and real business results. The first release should still limit users, volume, or scope while monitoring stays high. A pilot that performs well but has one or two incomplete areas needs additional operating evidence: named gaps and a real decision date, not an open-ended extension. A workflow lacking a business owner, reliable data, acceptable error limits, or measurable value needs redesign or retirement. A technically capable system can still be a poor business investment.
Before any of that gets decided, leadership needs a short evidence package in hand:
- The workflow map and the demand record.
- The named owners and the input profile.
- The evaluation report by error class.
- The review plan and the exception register.
- The security and monitoring controls.
- The user adoption record.
- The value report, comparing baseline against pilot results and full operating cost.
That package doubles as the starting documentation for whoever maintains the workflow after launch.
The first month is the test of whether the pilot survives contact with reality. Spend the first five days confirming owners, monitoring, alerts, and fallback procedures, and recording the approved baseline and performance limits. Release to a limited user group over the next five, reviewing outputs, exceptions, and corrections every day. Spend the following ten days comparing actual demand and user behavior against the pilot’s original assumptions. Resolve recurring problems through workflow changes rather than repeated manual rescue. Close out the month reviewing business outcomes, costs, and incidents, then decide whether to expand, revise, restrict, or stop the workflow.
A successful pilot proves an AI capability can contribute to a defined task. An operating workflow has to deliver that contribution repeatedly, across real people, real inputs, and real exceptions. Companies stuck in experimentation often keep examining the model when the real gap sits in the work surrounding it. A pilot has earned production when the organization can operate, govern, measure, maintain, and improve the complete workflow. That’s a different achievement than having built the part that made a good demo.
