by 
If you look back over the last 30 years, our technology history is riddled with failed projects, projects that never got off the ground, and projects that went massively over budget. And then there are the scandals.
The UK Post Office Horizon scandal stands as one of the most serious technology failures in modern British history. Between 1999 and 2015, around 1,000 sub-postmasters were wrongfully prosecuted after the Fujitsu-supplied Horizon accounting software recorded losses that did not exist. The total cost of redress now stands at around £2 billion. More than 13 people took their own lives. The technology did not just fail — it was trusted when it should have been questioned, and the consequences were devastating.
AI is just another technology. That is worth saying plainly. Although many of the possibilities being talked about are genuinely achievable, reality always kicks in. Every project — AI or otherwise — is a complicated combination of people, software, hardware, and process. That has been true for 30 years. It remains true now.
But we can always learn from the past. The question is whether we choose to.
AI is no different. And right now, the data on AI project failure should give everyone pause. The numbers are in. And they are not improving.
RAND Corporation, in one of the most rigorous independent analyses of AI project failure to date, interviewed 65 experienced data scientists and engineers across industries and company sizes. Their finding: more than 80% of AI projects fail to reach meaningful production deployment. That is twice the failure rate of IT projects without an AI component.
S&P Global’s 2025 survey of over 1,000 organisations across North America and Europe found that 42% of companies abandoned most of their AI initiatives that year. In 2024, that figure was 17%. The abandonment rate more than doubled in a single year. MIT’s Project NANDA, published in July 2025, found that 95% of organisations deploying generative AI saw zero measurable return. Not low return. Zero.
With global AI spending projected to reach $630 billion by 2028, these failure rates are not a statistic. They represent hundreds of billions of dollars in wasted investment, stalled initiatives, and businesses no closer to the outcomes they were promised.
What makes this harder to ignore is that the failure rate is moving in the wrong direction. The technology is more capable than it has ever been. The investment is larger than it has ever been. And yet more organisations are abandoning AI initiatives today than they were twelve months ago.
So what is actually going wrong?
The Research Points to Three Things
Across RAND, McKinsey, Gartner, MIT, and Informatica’s CDO Insights survey, the diagnosis is remarkably consistent. AI projects fail for three reasons, and they repeat themselves across industries, geographies, and organisation sizes.
The wrong use case. The process targeted for AI is not the right one, the problem definition is vague, or the initiative is chasing a technology rather than solving a business problem.
Data that is not ready. The data that would be needed to make AI work in production does not exist in the form required — it is fragmented, inconsistent, ungoverned, or simply not there.
A skills gap. The people needed to build, deploy, and sustain AI in a business context are not in place — and the organisation has not yet found a way to close that gap.
None of these are surprises. But the uncomfortable truth is that most organisations are still walking into all three of them, often simultaneously.
The Wrong Use Case: The Magpie Effect at Work
The first failure mode — choosing the wrong process to target — is something I have written about in depth. The Magpie Effect describes what happens when AI strategy is driven by possibility rather than process: the endless pivot towards the latest model, the newest capability, the most impressive vendor demo. Every pivot consumes time, burns budget, and erodes the momentum that comes from doing one thing properly and scaling it.
The RAND report is direct on this point. Stakeholders often misunderstand or miscommunicate what problem needs to be solved. Models get built and deployed optimised for the wrong metrics, or ones that simply do not fit into the real business workflow.
The antidote is starting with the Golden Process — the one business process that everything else hinges on — and asking what AI can do to remove the constraints within it. That conversation has to happen before any vendor is in the room, before any use case is evaluated, and certainly before any infrastructure is selected.
If you have not read the Magpie Effect post, the practical AI test at the end of it is a useful filter for any use case that lands on your desk.
Data That Is Not Ready: The Silent Failure
The second failure mode is the one that catches organisations by surprise — because it tends not to surface until a project is already in trouble.
The pattern is familiar. A proof of concept is built on a carefully selected, cleaned-up sample dataset. The demo runs well. Leadership approves production. And then everything stalls.
Production data is fragmented across systems that were never designed to talk to each other. Basic business terms — “customer”, “order”, “revenue” — are defined differently across departments. Historical records have gaps. Formatting is inconsistent. The clean sample that powered the demo bears almost no resemblance to the messy reality of how the business actually runs.
Informatica’s 2025 CDO Insights survey found that data quality and readiness was the top obstacle to AI success, cited by 43% of organisations. McKinsey found that organisations reporting significant AI returns are twice as likely to have invested in data infrastructure before selecting modelling techniques. Gartner predicts that 60% of AI projects lacking AI-ready data will be abandoned entirely.
This is not a new problem. It is the same problem that has existed since organisations first tried to build anything useful on top of their data. The difference is that AI amplifies it. A flawed report can be corrected. A model trained on broken foundations will confidently produce broken outputs at scale — and the damage compounds before anyone notices.
I have covered the data readiness problem in detail in Garbage In, Expensive Garbage Out and A Million Rows of Nothing. The short version: AI does not fix bad data. It scales it.
The Skills Gap: The Barrier Nobody Budgets For
The third failure mode is the one that receives the least attention — and may be the hardest to solve quickly.
The last 30 years of enterprise IT have built deep, hard-won expertise in infrastructure. Server architecture. Storage design. Networking and virtualisation. Security and compliance. That expertise is not obsolete — it remains essential. The physical and virtual foundations that AI runs on still need people who understand them properly.
But AI demands a different and additional skill set that most organisations are still building.
| Traditional IT Skills (Still Relevant) | New Skills Now Required |
|---|---|
| Server architecture and management | Data engineering |
| Storage design and optimisation | AI/ML engineering |
| Networking and virtualisation | Data science |
| Security and compliance | Business domain knowledge |
McKinsey’s 2024 State of AI survey found that 58% of businesses are hampered by internal AI skill shortages. Informatica’s CDO Insights survey placed skills and data literacy third in the list of top AI obstacles. PwC found that almost 65% of executives acknowledge their AI initiatives are not succeeding because of a lack of executive sponsorship — a leadership gap that is itself a symptom of not having people in the room who can translate between business outcomes and AI capability.
The skills gap is not a technology problem. It is a people and partnership problem. And it is the reason most AI pilots never leave the room they were born in.
Picture a typical scenario. An organisation identifies a strong AI use case, aligned to a real business process. The data is in reasonable shape. Leadership is supportive. A vendor is engaged. And then the questions start: who is going to build the data pipeline? Who owns the model in production? Who bridges the gap between what the model does and what the business actually needs it to do? The room goes quiet. Not because the will isn’t there — but because the people aren’t.
The honest message is that organisations do not need to replace their existing IT teams — they need to extend them. The people who understand the infrastructure are still essential. But they need colleagues who understand data pipelines, model behaviour, and the business processes those models need to serve. That combination is rare. It takes time to build. And in its absence, even the right use case with clean data will stall.
This is why Partner selection matters as much as use case selection. The right Partner does not just bring technical capability — they bring the scars of what does not work. An AI and data practice built over years has already made the mistakes most organisations have not made yet. That is not a credential. That is insurance.
AI is not a product, it’s an eco-system built on partnership
Three Causes. One Pattern.
What the research is describing — even when it does not use these words — is organisations that jumped to the model before they had solved the three problems that sit upstream of it.
They chased the possibility rather than the process. They skipped the data foundations. And they underestimated how different the skills requirement would be.
McKinsey’s summary of what separates the 6% of genuine AI high performers from everyone else is as sharp as it gets: AI is 20% algorithms, 80% organisational rewiring.
The organisations building durable AI capability are not necessarily the ones with the most sophisticated models. They are the ones that got the process right, made the data ready, and put the right people in place — before they wrote a single line of model code.
The failure statistics look alarming until you understand the causes. Then they look entirely predictable.
The good news: all three are solvable. None of them require waiting for the next frontier model. They just require doing the less glamorous work first.
I have written about what that looks like in practice. Getting AI Right First Time sets out a five-step path from honest awareness through to durable, enterprise-wide capability — the journey from AI as a science project to AI as part of how the business actually runs. And In Praise of Boring, Everyday AI makes the case for what success actually looks like in production: not the demo, not the headline, but the quiet system that runs on a Tuesday afternoon without anyone noticing — because it has simply become part of how the business works.
That is the goal. Not AI that impresses. AI that endures.
AI success has to be built on the right foundations — not retrofitted onto broken ones.
Leave a Reply