by Why the AI industry’s data problem isn’t an accident
I’ve been fascinated by data ever since I connected SQL Server, via IBM Client Access, to the AS/400 at Volvo Trucks UK. Suddenly I could query data, visualise data and report facts in near real time through ASP.NET web pages. Rows of numbers, product names, dates and times, all queried, sorted and presented so they made sense. Everything comes from data.
But here’s my observation. Because data engineering takes skill, business acumen and patience, data is often left behind, the poor cousin of the more exciting AI models.
In my recent studies I came across a paper that confirms my thinking. Its title says it all: “Everyone wants to do the model work, not the data work.” This post was inspired by that paper and by that way of thinking, because to understand data is to understand AI.
That paper, Sambasivan et al. (2021), published at CHI, contains a finding that should have stopped the AI industry in its tracks.
Ninety-two percent of AI practitioners working in high-stakes domains reported experiencing at least one data cascade. The domains were health, conservation, road safety and credit. A data cascade is an upstream data problem, ignored or undetected, that compounds quietly through annotation, training and evaluation before surfacing, expensively, in production.
Ninety-two percent. In domains where the downstream consequences could include missed cancer diagnoses, poaching locations that went unflagged and road safety systems that failed to perform.
In Now You See It, Now You Don’t, I looked at what a data cascade is. Now I want to look at why the industry keeps producing them. I’ve looked at what a data cascade is. Now I want to look at why the industry keeps producing them.
This Isn’t a Competence Problem
The first thing to understand is that the practitioners in this study were not careless. They were experienced and technically capable, and the paper is explicit that they were deeply committed to the communities their systems were meant to serve. They understood that data quality mattered. They said so, repeatedly, in the interviews.
And yet 92% of them experienced data cascades anyway.
That’s not a competence problem. It’s a system problem.
Nobody sat down and designed it this way, of course. Systems don’t need a designer to produce consistent results. They produce whatever their incentives reward, and the incentives around AI reward the model.
The Prestige Economy
Academic AI runs on publications. Publications get cited when they introduce novel model architectures, beat benchmarks, or demonstrate new capabilities. A paper that carefully documents a dataset, audits its representativeness and identifies its limitations generates far fewer citations than one that tops a benchmark leaderboard. Data just isn’t sexy enough.
This isn’t irrational behaviour. People are responding correctly to the reward system they’re in. As Eli Goldratt put it: “Tell me how you measure me, and I will tell you how I will behave.” The problem is what the reward system measures: novelty and performance on controlled benchmarks, not robustness in the real world.
That academic culture flows directly into industry. The engineers who get hired, promoted and celebrated are the ones who shipped the model, not the ones who spent three months auditing the training data. The data work is largely invisible. It produces no demo and no benchmark score, and it’s very hard to put in a performance review.
“Everyone wants to do the model work, not the data work.” — P4, healthcare practitioner, India.
The Grunt Work Framing
There’s a deeper cultural problem. Data work has been framed as the unglamorous precursor to the real work, rather than as a discipline in its own right.
This framing has a history. Consider ImageNet. Its dataset made the deep learning breakthrough possible, yet the celebrated achievement was AlexNet, the model trained on it. The same pattern followed with the landmark systems that came after: the models became the headlines and the data became the infrastructure.
“Garbage in, garbage out” is a phrase everyone knows. Yet the field was slow to build anything like software engineering’s code review culture around data. Useful tools have emerged in recent years, including Datasheets for Datasets, dedicated dataset tracks at major conferences and the broader data-centric AI movement. But outside a minority of mature organisations, data review is still not standard practice in production. The vocabulary for talking about data quality is still largely borrowed from database management.
The result is that data work is treated as a cost to be minimised, not a capability to be built.
The Visibility Problem
Model failures are visible. A model that misclassifies something produces a wrong output that you can see, measure and attribute.
A data problem is different. Consider a dataset that systematically underrepresents a subpopulation, a capture rig that was never validated for the task it was assigned, or an annotation scheme that embedded a subjective assumption nobody documented. None of these announce themselves. They produce a model that looks fine in evaluation and fails in production, and the failure looks like a model problem.
Cascades are opaque by nature. The signal arrives at the wrong end of the pipeline, long after the upstream decision that caused it. In the study, some cascades took two to three years to manifest. By then, the people who made the data decisions are often not the people who discover the failure, and often not the people who pay for it.
Nobody gets fired for a data decision made eighteen months ago that only surfaced in production last week. The accountability is diffuse, so the incentive to get it right in the first place is correspondingly weak.
The Expectation Problem
Customers expect magic, and to be fair to them, the industry has spent years telling them to.
Buyers want a high-performing model. Few want to fund six months of data collection, annotation and quality auditing before a single model is trained. Suppliers, keen to win the work, are rarely incentivised to insist on it. The pressure to demonstrate results quickly compresses the time available for data work, whether that means showing something in a pilot or hitting a proof-of-concept milestone. It also creates incentives to skip the data work altogether.
This is where the cascade often starts. It usually isn’t negligence but a response to commercial pressure on both sides of the table. The data work gets deferred. The pilot looks good. The contract gets signed. The production deployment reveals what the pilot data never showed.
The practitioners in the study described clients expecting “magic” from AI, with high performance thresholds set without much thought for the underlying data or process. Some admitted to hacking model performance for client demonstrations, inflating accuracy to meet expectations while knowing the underlying data wasn’t ready.
That doesn’t excuse it. But it does show what happens when the incentive to show results overrides the incentive to get it right.
The Education Gap
Most AI education, even at graduate level, is built around clean, pre-labelled benchmark datasets. Students learn to train models on data that already exists, is already clean and is already representative of the task. Many graduate without ever having designed a data collection process, written an annotation guide, audited a dataset for distributional bias, or dealt with live data drifting away from training conditions.
As one practitioner in the study put it, real-world data is never clean, yet courses focus on models and tools and rarely teach data cleaning or pipeline gaps.
So when graduates enter industry and meet a messy, incomplete, biased real-world dataset, they don’t have the tools to diagnose it. They do what they were trained to do: tune the model.
I’ve seen that instinct in my own work. When I scaled up the neuron counts in my Neo-ML experiments, accuracy didn’t move. Only the memory footprint grew. More model bought me nothing, because the ceiling was set by the data.
The education system produces practitioners who are technically excellent at the wrong layer of the problem.
The Goodness-of-Fit Trap
Current AI evaluation is almost entirely metrics-driven: accuracy, F1, AUC. These are measured on held-out data, but that data is usually drawn from the same source, under the same conditions, as the training set. They tell you how well the model fits that slice of the world. They say nothing about whether that slice was the right one.
A model can achieve 97% accuracy on an evaluation set that shares the training set’s biases. It can then fail immediately in production when it meets the part of the world the dataset forgot to represent. The metrics looked fine. The data wasn’t.
The paper calls this the goodness-of-fit trap. It argues for a shift toward goodness-of-data metrics, meaning measures of whether a dataset actually represents the phenomenon it claims to capture. No standardised metrics for this currently exist.
We are measuring the wrong thing, with the wrong tools, at the wrong point in the process. And then we’re surprised when the system fails.
What This Looks Like in Practice
You don’t need to be building a cancer detection model for this to apply to you.
Picture a winery that deploys a defect detection model trained almost entirely on daytime footage, then wonders why it underperforms on the night shift. Or a distribution centre that evaluates its model against a dataset carrying the same seasonal bias as its training data, declares success and ships. Or a computer vision system that passes evaluation at 97% accuracy and fails in production because its evaluation set was drawn from the same narrow conditions as its training set. Every one of these is a data cascade.
The mechanism is identical. Only the stakes differ. That winery, as it happens, is a problem we’ll be returning to.
The Honest Structural Answer
The deepest reason data cascades are so prevalent is that data quality is a collective action problem.
The benefits of good data accrue to the whole system: the model, the deployment, the end user and the organisation. But the costs fall on individuals who get no personal credit for doing the work well, and who face real career costs for slowing down to do it properly.
Until the field builds reward structures that make data excellence visible, measurable and career-relevant, the rational individual response will continue to be to do the model work, not the data work. Code quality, system reliability and model performance are all rewarded this way; data excellence needs to be too.
The paper doesn’t say practitioners don’t care. It says they care deeply, and the system overrides them anyway.
The Fix
The paper’s conclusion is structural, not technical. Individual practitioners can try to avoid data cascades in their own work. But without systemic change in how data work is rewarded, recognised and resourced, the 92% figure won’t move.
That change starts with conferences and publications treating dataset documentation, provenance and representativeness as expected rather than optional. Some now have dedicated tracks for this; the norm needs to follow. It continues in organisations that reward data collection, annotation quality and pipeline maintenance in promotions and peer reviews, the same way good software engineering is rewarded. It needs education that teaches data design, collection and quality as core skills rather than afterthoughts. And it needs customers and suppliers alike to accept that the data work is the work, not the overhead before the work starts.
Before anyone asks “which model?”, someone should be asking “what does the dataset actually represent?” That means auditing the coverage, checking the conditions and mapping the annotation assumptions. The model inherits everything the data got right or wrong, and no amount of architectural sophistication buys that back.
None of this is technically complex. All of it requires treating data as a first-class citizen of the AI development process.
The system produced 92% data cascade prevalence because its incentives, its education, its reward structures and its expectations all pointed that way. In that sense, it is working exactly as designed. It will keep producing that result until the design changes.
To understand data is to understand AI. It’s time the industry started rewarding the people who do.
Reference: Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P. and Aroyo, L. (2021) ‘”Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI’, Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–15. https://doi.org/10.1145/3411764.3445518
Leave a Reply