Posted on

To Understand Data is to Understand AI

Why the AI industry’s data problem isn’t an accident

Blueprint-style illustration depicting a model processing data, with stages labeled: Annotation, Training, Evaluation, and Production, highlighted by a lightning bolt effect.

I’ve been fascinated by data ever since I connected SQL Server, via IBM Client Access, to the AS/400 at Volvo Trucks UK. Suddenly I could query data, visualise data and report facts in near real time through ASP.NET web pages. Rows of numbers, product names, dates and times, all queried, sorted and presented so they made sense. Everything comes from data.

But here’s my observation. Because data engineering takes skill, business acumen and patience, data is often left behind, the poor cousin of the more exciting AI models.

In my recent studies I came across a paper that confirms my thinking. Its title says it all: “Everyone wants to do the model work, not the data work.” This post was inspired by that paper and by that way of thinking, because to understand data is to understand AI.

That paper, Sambasivan et al. (2021), published at CHI, contains a finding that should have stopped the AI industry in its tracks.

Ninety-two percent of AI practitioners working in high-stakes domains reported experiencing at least one data cascade. The domains were health, conservation, road safety and credit. A data cascade is an upstream data problem, ignored or undetected, that compounds quietly through annotation, training and evaluation before surfacing, expensively, in production.

Ninety-two percent. In domains where the downstream consequences could include missed cancer diagnoses, poaching locations that went unflagged and road safety systems that failed to perform.

In Now You See It, Now You Don’t, I looked at what a data cascade is. Now I want to look at why the industry keeps producing them. I’ve looked at what a data cascade is. Now I want to look at why the industry keeps producing them.

This Isn’t a Competence Problem

The first thing to understand is that the practitioners in this study were not careless. They were experienced and technically capable, and the paper is explicit that they were deeply committed to the communities their systems were meant to serve. They understood that data quality mattered. They said so, repeatedly, in the interviews.

And yet 92% of them experienced data cascades anyway.

That’s not a competence problem. It’s a system problem.

Nobody sat down and designed it this way, of course. Systems don’t need a designer to produce consistent results. They produce whatever their incentives reward, and the incentives around AI reward the model.

The Prestige Economy

Academic AI runs on publications. Publications get cited when they introduce novel model architectures, beat benchmarks, or demonstrate new capabilities. A paper that carefully documents a dataset, audits its representativeness and identifies its limitations generates far fewer citations than one that tops a benchmark leaderboard. Data just isn’t sexy enough.

This isn’t irrational behaviour. People are responding correctly to the reward system they’re in. As Eli Goldratt put it: “Tell me how you measure me, and I will tell you how I will behave.” The problem is what the reward system measures: novelty and performance on controlled benchmarks, not robustness in the real world.

That academic culture flows directly into industry. The engineers who get hired, promoted and celebrated are the ones who shipped the model, not the ones who spent three months auditing the training data. The data work is largely invisible. It produces no demo and no benchmark score, and it’s very hard to put in a performance review.

“Everyone wants to do the model work, not the data work.” — P4, healthcare practitioner, India.

The Grunt Work Framing

There’s a deeper cultural problem. Data work has been framed as the unglamorous precursor to the real work, rather than as a discipline in its own right.

This framing has a history. Consider ImageNet. Its dataset made the deep learning breakthrough possible, yet the celebrated achievement was AlexNet, the model trained on it. The same pattern followed with the landmark systems that came after: the models became the headlines and the data became the infrastructure.

“Garbage in, garbage out” is a phrase everyone knows. Yet the field was slow to build anything like software engineering’s code review culture around data. Useful tools have emerged in recent years, including Datasheets for Datasets, dedicated dataset tracks at major conferences and the broader data-centric AI movement. But outside a minority of mature organisations, data review is still not standard practice in production. The vocabulary for talking about data quality is still largely borrowed from database management.

The result is that data work is treated as a cost to be minimised, not a capability to be built.

The Visibility Problem

Model failures are visible. A model that misclassifies something produces a wrong output that you can see, measure and attribute.

A data problem is different. Consider a dataset that systematically underrepresents a subpopulation, a capture rig that was never validated for the task it was assigned, or an annotation scheme that embedded a subjective assumption nobody documented. None of these announce themselves. They produce a model that looks fine in evaluation and fails in production, and the failure looks like a model problem.

Cascades are opaque by nature. The signal arrives at the wrong end of the pipeline, long after the upstream decision that caused it. In the study, some cascades took two to three years to manifest. By then, the people who made the data decisions are often not the people who discover the failure, and often not the people who pay for it.

Nobody gets fired for a data decision made eighteen months ago that only surfaced in production last week. The accountability is diffuse, so the incentive to get it right in the first place is correspondingly weak.

The Expectation Problem

Customers expect magic, and to be fair to them, the industry has spent years telling them to.

Buyers want a high-performing model. Few want to fund six months of data collection, annotation and quality auditing before a single model is trained. Suppliers, keen to win the work, are rarely incentivised to insist on it. The pressure to demonstrate results quickly compresses the time available for data work, whether that means showing something in a pilot or hitting a proof-of-concept milestone. It also creates incentives to skip the data work altogether.

This is where the cascade often starts. It usually isn’t negligence but a response to commercial pressure on both sides of the table. The data work gets deferred. The pilot looks good. The contract gets signed. The production deployment reveals what the pilot data never showed.

The practitioners in the study described clients expecting “magic” from AI, with high performance thresholds set without much thought for the underlying data or process. Some admitted to hacking model performance for client demonstrations, inflating accuracy to meet expectations while knowing the underlying data wasn’t ready.

That doesn’t excuse it. But it does show what happens when the incentive to show results overrides the incentive to get it right.

The Education Gap

Most AI education, even at graduate level, is built around clean, pre-labelled benchmark datasets. Students learn to train models on data that already exists, is already clean and is already representative of the task. Many graduate without ever having designed a data collection process, written an annotation guide, audited a dataset for distributional bias, or dealt with live data drifting away from training conditions.

As one practitioner in the study put it, real-world data is never clean, yet courses focus on models and tools and rarely teach data cleaning or pipeline gaps.

So when graduates enter industry and meet a messy, incomplete, biased real-world dataset, they don’t have the tools to diagnose it. They do what they were trained to do: tune the model.

I’ve seen that instinct in my own work. When I scaled up the neuron counts in my Neo-ML experiments, accuracy didn’t move. Only the memory footprint grew. More model bought me nothing, because the ceiling was set by the data.

The education system produces practitioners who are technically excellent at the wrong layer of the problem.

The Goodness-of-Fit Trap

Current AI evaluation is almost entirely metrics-driven: accuracy, F1, AUC. These are measured on held-out data, but that data is usually drawn from the same source, under the same conditions, as the training set. They tell you how well the model fits that slice of the world. They say nothing about whether that slice was the right one.

A model can achieve 97% accuracy on an evaluation set that shares the training set’s biases. It can then fail immediately in production when it meets the part of the world the dataset forgot to represent. The metrics looked fine. The data wasn’t.

The paper calls this the goodness-of-fit trap. It argues for a shift toward goodness-of-data metrics, meaning measures of whether a dataset actually represents the phenomenon it claims to capture. No standardised metrics for this currently exist.

We are measuring the wrong thing, with the wrong tools, at the wrong point in the process. And then we’re surprised when the system fails.

What This Looks Like in Practice

You don’t need to be building a cancer detection model for this to apply to you.

Picture a winery that deploys a defect detection model trained almost entirely on daytime footage, then wonders why it underperforms on the night shift. Or a distribution centre that evaluates its model against a dataset carrying the same seasonal bias as its training data, declares success and ships. Or a computer vision system that passes evaluation at 97% accuracy and fails in production because its evaluation set was drawn from the same narrow conditions as its training set. Every one of these is a data cascade.

The mechanism is identical. Only the stakes differ. That winery, as it happens, is a problem we’ll be returning to.

The Honest Structural Answer

The deepest reason data cascades are so prevalent is that data quality is a collective action problem.

The benefits of good data accrue to the whole system: the model, the deployment, the end user and the organisation. But the costs fall on individuals who get no personal credit for doing the work well, and who face real career costs for slowing down to do it properly.

Until the field builds reward structures that make data excellence visible, measurable and career-relevant, the rational individual response will continue to be to do the model work, not the data work. Code quality, system reliability and model performance are all rewarded this way; data excellence needs to be too.

The paper doesn’t say practitioners don’t care. It says they care deeply, and the system overrides them anyway.

The Fix

The paper’s conclusion is structural, not technical. Individual practitioners can try to avoid data cascades in their own work. But without systemic change in how data work is rewarded, recognised and resourced, the 92% figure won’t move.

That change starts with conferences and publications treating dataset documentation, provenance and representativeness as expected rather than optional. Some now have dedicated tracks for this; the norm needs to follow. It continues in organisations that reward data collection, annotation quality and pipeline maintenance in promotions and peer reviews, the same way good software engineering is rewarded. It needs education that teaches data design, collection and quality as core skills rather than afterthoughts. And it needs customers and suppliers alike to accept that the data work is the work, not the overhead before the work starts.

Before anyone asks “which model?”, someone should be asking “what does the dataset actually represent?” That means auditing the coverage, checking the conditions and mapping the annotation assumptions. The model inherits everything the data got right or wrong, and no amount of architectural sophistication buys that back.

None of this is technically complex. All of it requires treating data as a first-class citizen of the AI development process.

The system produced 92% data cascade prevalence because its incentives, its education, its reward structures and its expectations all pointed that way. In that sense, it is working exactly as designed. It will keep producing that result until the design changes.

To understand data is to understand AI. It’s time the industry started rewarding the people who do.

Reference: Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P. and Aroyo, L. (2021) ‘”Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI’, Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–15. https://doi.org/10.1145/3411764.3445518

Posted on

When the Wine Lies: Finding Quality in the Chemistry

Illustration of a wine bottle with measurements and analysis labels including alcohol, volatile acidity, sulphates, pH, and density, alongside a scatter plot showing data points with ratings.

Sometimes the data tells you what your wallet couldn’t.

Last weekend I cooked steak. Proper steak — the kind that deserves a decent red wine alongside it. So I did what any self-respecting wine buyer does: I spent more than usual. Higher price, better wine. That’s how it works, right?

Wrong.

The wine was awful. Bitter, sharp, aggressive — more paint stripper than Pinot. The kind of wine that makes you wonder whether the person who priced it had ever actually tasted it. As I pushed the glass aside and reached for the water, a question formed: in the real world, how do we actually measure wine quality?

Not price. Clearly not price. So what then?

Meet Charlie and Clare

Regular readers will know Charlie and Clare. Charlie is our Data Engineer — he builds the pipelines, aggregates the sources, and delivers clean, structured data. Clare is our Data Analyst and Data Scientist — she works with what Charlie hands her and finds the patterns worth knowing about.

This week they’re working with wine.

Charlie has been busy. A winery client wants to understand quality across their production batches. The data lives in multiple places — laboratory analysis systems, fermentation monitoring sensors, batch production records. Individually, each source is a fragment. Together, they tell a story. Charlie’s data platform pulls from all of them, normalises the formats, handles the joins, and delivers Clare a clean pipeline: 1,599 red wine samples, each described by eleven physicochemical measurements.

Alcohol content. Fixed and volatile acidity. Citric acid. Sulphates. Residual sugar. Chlorides. Density. pH. Free and total sulphur dioxide.

Eleven numbers per wine.

And crucially — no quality labels.

Clare’s Problem

Clare looks at the dataset and faces a genuinely interesting challenge. There are no categories here, no pre-assigned groups, no right answers to train against. Just eleven continuous measurements and 1,599 rows of chemistry.

This is the domain of unsupervised learning — the branch of machine learning that finds structure in data without being told what to look for. Where supervised learning optimises toward a target, unsupervised learning asks a different question entirely: what patterns exist that we haven’t defined yet?

Clare’s task is to let the data organise itself, then ask whether the organisation means anything.

Before she touches a model, she does something essential. She scales the data.

This matters more than it sounds. The eleven features live on very different scales — alcohol ranges from roughly 8 to 15, total sulphur dioxide from 6 to 289. Feed raw numbers into a distance-based algorithm and the large-scale variables will dominate purely through magnitude, drowning out the signal from smaller-range features. StandardScaler transforms everything to zero mean and unit variance — now every feature competes on equal terms.

Charlie’s pipeline has already handled the missing values and format inconsistencies. Clare inherits clean data. That’s not an accident — it’s the platform working as intended.

Eleven Dimensions Are Too Many to See

Before clustering, Clare reduces complexity using Principal Component Analysis — PCA.

Think of PCA as finding the angles from which the data is most spread out. Eleven features create eleven dimensions, which is impossible to visualise and cognitively overwhelming to reason about. PCA finds new axes — principal components — that capture the maximum variance in the fewest dimensions.

The results are telling. Nine components are needed to explain 95% of the variance. No single axis dominates. The data genuinely is high-dimensional — there’s no shortcut that captures most of the story. The first two components together explain just 45.7% of variance.

That’s an important caveat Clare keeps front of mind: when she later visualises clusters on a two-dimensional PCA plot, she’s seeing less than half the structure. The scatter plot is illustrative, not definitive.

PC1 is driven primarily by acidity-related features — it broadly separates sharper, more acidic wines from rounder ones. PC2 captures alcohol and fermentation character — higher alcohol and sulphate concentrations, reflecting more complete fermentation and stronger microbial stability. Even this compressed view starts to suggest that wine chemistry has meaningful directions of variation.

How k-means Works — The Wine Tasting Table

Imagine Clare pours all 1,599 wine samples into glasses and lines them up on a long tasting table. She doesn’t know how many groups there are yet, but she suspects wines with similar chemistry will naturally belong together.

She picks three glasses to act as reference points — her starting cluster centres, her “k” — and assigns every other wine to whichever reference glass it’s closest to, chemically speaking. Then she looks at each group, finds the glass that sits closest to the average of all its members, and moves her reference point there. Wines get reassigned. Reference points shift again. The process repeats until nothing moves anymore — the groups have stabilised around their natural centres.

That’s k-means. Not magic, not mystery. An algorithm that keeps nudging reference glasses along the table until the groupings settle. The “k” is simply the number of reference points — Clare’s job is to choose it wisely, which is where the elbow method comes in.

Letting the Data Find Its Own Groups

Clare runs k-means clustering — an algorithm that partitions observations into k groups by minimising the distance between each point and its cluster centre.

The question is: what should k be?

She uses two methods in parallel. The elbow method plots inertia — the total within-cluster variance — against increasing values of k. As k grows, inertia falls; the question is where the rate of improvement flattens. The silhouette coefficient measures how well each point sits within its assigned cluster compared to neighbouring clusters — higher is better.

Both methods point toward k=3 as a defensible choice. Three groups. Clare fits the final model.

The clusters contain 722, 502, and 375 wines respectively.

What the Clusters Actually Found

Now Clare looks at the chemistry of each group.

Cluster 1 — 502 wines — stands out immediately. Highest alcohol (10.72%), lowest volatile acidity (0.41 g/dm³), highest sulphates (0.75 g/dm³). These are markers experienced winemakers recognise: lower volatile acidity means less acetic acid, less of that sharp, vinegary edge. Higher sulphates support microbial stability and structure.

Cluster 0 — 722 wines — shows the inverse pattern. Highest volatile acidity (0.61), lowest sulphates (0.61). More of that aggressive sharpness Clare’s colleague experienced at the weekend.

Cluster 2 — 375 wines — is characterised by elevated sulphur dioxide levels and the lowest alcohol of the three groups (9.88%), suggesting less complete fermentation.

Three chemical profiles. Found without a single quality label in sight.

The Reveal

Now Clare looks at quality.

She takes the quality scores — held back throughout the entire analysis — and calculates the mean score per cluster. This is post-hoc interpretation only. The clustering didn’t know about quality. But the results are striking.

ClusterProfileMean Quality
1High alcohol, low volatile acidity, high sulphates5.96
0High volatile acidity, lower sulphates5.55
2High sulphur dioxide, lowest alcohol5.36

The algorithm — working only from chemistry — has separated wines in a way that aligns meaningfully with human quality judgement. The group with the most favourable chemical profile scores highest. The group with the most aggressive volatile acidity scores lowest.

Clare didn’t tell the model what quality meant. The chemistry already knew.

An Honest Number

The silhouette score is 0.19.

By textbook standards that’s weak. Some analysts would look at that number and worry. Clare doesn’t, and it’s worth understanding why.

Wine chemistry is continuous. There are no hard walls between a quality-6 wine and a quality-7 wine — no moment where one chemical compound crosses a threshold and suddenly the wine is better. The boundaries between clusters are gradual, overlapping, real-world messy. A low silhouette score in this context isn’t a sign that the analysis failed. It’s information about the nature of the data itself.

The clusters are soft. The patterns are genuine. These two things are not contradictory.

This matters for how Clare reports her findings. She isn’t presenting three neat buckets — “good wine, average wine, poor wine.” She’s presenting three chemical tendencies, with meaningful separations on the features that wine science already tells us matter.

Why Charlie’s Platform Made This Possible

It’s worth pausing on something easy to take for granted.

Clare’s analysis worked because she had complete, clean, comparable data. Every one of the 1,599 samples described by the same eleven features, scaled and pipeline-ready.

In the real world, that’s rarely the starting point. Laboratory analysis lives in one system. Sensor data from fermentation monitoring lives in another. Batch and production records in a third. Pricing and commercial data somewhere else entirely. Each system uses different formats, different naming conventions, different update frequencies.

Without Charlie’s data platform aggregating those sources into a coherent, governed pipeline, Clare isn’t doing unsupervised learning on 1,599 wines. She’s manually reconciling spreadsheets and hoping nothing got lost in the joins.

The insight — that chemical profile predicts perceived quality independently of price — is only discoverable when the data foundation exists to support the question. Structure has to be built in, not bolted on.

What Clare Does Next

Unsupervised learning is Clare’s first move with an unfamiliar dataset. It reveals what’s there before asking what predicts what.

The natural next step is supervised learning. Now that three chemical profiles have been identified, Clare can use cluster membership to inform stratified sampling — ensuring any training dataset for a quality prediction model includes representative coverage of all three groups rather than accidentally over-representing one chemical type.

She could also bring price into the analysis. If Charlie’s platform connects to commercial data, Clare can ask the question that started this whole investigation: does chemical profile correlate with price? Is the expensive-but-terrible wine an outlier, or is the price-quality assumption systematically weak across the portfolio?

That’s a question worth answering before anyone’s next steak dinner.

The Takeaway

Eleven numbers. No labels. Three meaningful groups.

Clare found chemical structure that aligns with human quality judgement — not because she told the algorithm what quality meant, but because the chemistry already encoded it. Unsupervised learning didn’t give her answers. It gave her the right questions.

And behind all of it, doing the unglamorous work that makes the glamorous work possible, was Charlie’s data platform.

Quality has to be found in the data. But first, the data has to be there to find it.

Next in the series: Clare takes the cluster profiles into supervised learning — and finds out whether chemistry can predict quality well enough to save the rest of us from expensive mistakes.

Posted on

A Million Rows of Nothing

A graphic illustrating a grid labeled 'A MILLION ROWS OF NOTHING,' featuring numerical values, with most cells showing '0.00' and select cells highlighted in orange displaying '1.00.' A crossed-out server icon is on the left, and a note at the bottom reads 'DO NOT SKIP.'

Why business use case and data strategy must come before AI strategy

At a customer event last year, an IT Director told me — with some confidence — that they already had an AI strategy.

“Great,” I said. “Now tell me about your data strategy.”

“We have 250TB,” he replied.

I nodded. And thought: there is a very big difference between data and storage.

That moment has stayed with me because it wasn’t an isolated conversation. It was a pattern. Organisations are arriving at the AI table with infrastructure plans, vendor commitments and boardroom ambition — but without first validating the business use case, the predicted ROI, or the data required to support either one.

That is the gap. And it is an expensive one.

The gap is earlier than most organisations think

Walk any AI conference floor and the energy is real. The technology is genuinely impressive. GPU servers are being specced, procured and racked. Data scientists are being hired. AI roadmaps are being presented to boards.

And somewhere near the bottom of the slide deck, almost as an afterthought: “We’ll need to look at data readiness.”

For organisations serious about AI delivering real outcomes, this is the wrong order.

The first question should not be “what infrastructure should we buy?” It should be “what business problem are we solving, what return do we expect, and does our data actually support that outcome?”

If those questions haven’t been answered, the AI strategy isn’t yet a strategy. It’s an ambition.

Start with the business use case and predicted ROI

Before talking about models or servers, organisations need clarity on three things: what specific business problem are we trying to solve, what result would make this investment worthwhile, and what evidence suggests the data can support that result?

This matters because businesses don’t invest in AI for the sake of AI. They invest in outcomes — lower cost, higher revenue, reduced risk, better service, faster decisions, improved productivity.

The business use case and predicted ROI have to come first. They set the standard the data must meet, the model must prove, and the infrastructure must eventually support. Without that anchor, teams end up building technical capability in search of commercial justification.

Then comes data strategy

This is where many organisations confuse capacity with capability.

Saying “we have 250TB” is not describing a data strategy. It is describing a storage estate.

A real data strategy answers different questions. What data actually matters for the use case? Where does it live? Who owns it? How is it governed? How trustworthy is it? How easily can it be accessed, joined, prepared and used?

AI doesn’t begin with infrastructure. It begins with understanding whether the organisation has data that is usable, relevant, governed and connected to a business objective. That is why data strategy has to come before AI strategy. If you don’t understand the asset you’re asking AI to learn from, you don’t yet know whether the strategy is viable.

Data engineering is not pre-work. It is the work.

The foundational argument is simple, even if it’s routinely ignored: data engineering is not a precursor to AI work. It is AI work.

The pipelines, the schemas, the quality checks, the lineage, the transformation logic — these are not the boring bit before the interesting bit starts. They are the work.

A model is only ever as good as the data it learns from. If that data is incomplete, inconsistently formatted, poorly labelled or structurally flawed, the model will learn the wrong things with great efficiency. Garbage in, amplified garbage out at scale.

The data engineering layer needs to be in place — and understood — before a model is trusted in production. That means clean, documented pipelines with known lineage, a clear system of record for the domain you’re working in, variables that are what they say they are, and critically — someone who has actually interrogated the data, not just counted the rows.

250TB of storage tells you nothing about any of that.

Even clean data can still be useless

Here is where the conversation gets more uncomfortable. Because the problem isn’t always dirty data.

Sometimes the data looks clean. The schema is tidy. The row counts are impressive. The formatting is consistent. It passes the hygiene checks. And then you run the analysis — and discover the data tells you very little. Not because it’s messy. Because it’s empty of useful signal.

This is the moment EDA — Exploratory Data Analysis — earns its place. Not as a technical formality, not as a box to tick before the real work starts, but as the moment of truth. The point at which you find out whether your data can actually answer the question you’re asking of it.

That means looking at distributions, missingness, outliers, feature relationships, basic correlations, and whether the patterns you expected to see are actually present. If they aren’t, that isn’t a minor issue. It’s the whole issue.

A million rows of nothing is still nothing

This is why volume can be so misleading.

Take a look at this correlation matrix.

A correlation matrix displaying the relationships among numerical features including Price, Discount, Tax Rate, Stock Level, Customer Age Group, Shipping Cost, Return Rate, Seasonality, and Popularity Index. The matrix is color-coded with a gradient scale from blue to red indicating strength of correlation.

To the untrained eye it looks impressive. Professional. The kind of output that gets nodded at in a boardroom. But look closer. That red diagonal? Every variable correlating perfectly with itself — mathematically guaranteed, analytically meaningless. Everything else is zero. Price and Discount: no relationship. Seasonality and Stock Level: no relationship. Shipping Cost and Return Rate: no relationship.

In a real retail dataset those relationships should exist. The fact that this data shows none of them is a signal worth taking seriously. A flat correlation view doesn’t prove there is absolutely nothing to learn — but it does tell you there is no obvious predictive signal in this view of the data. That should trigger caution, not confidence.

You shouldn’t respond by buying more infrastructure. You should respond by asking better questions. Are these the right features? Is the data aggregated at the wrong level? Are important variables missing? Is the business question badly framed? Are we trying to predict something the data cannot meaningfully support?

If you can’t answer those questions, you are not ready to build the model. You are ready to do more analysis.

Model readiness comes after data readiness

Only once the business case is clear and the data has been tested should the conversation move to model readiness.

At that point the focus becomes more disciplined. Can the data support the target outcome? Which features actually carry useful predictive weight? What baseline performance is realistic? What error level is acceptable for the business use case? What would success look like in practice, not just in a notebook?

This is the stage where organisations find out whether the use case is genuinely model-worthy — or whether it looked better in a strategy deck than it does in reality. Model readiness is not about enthusiasm. It is about proof.

Infrastructure should be the consequence, not the starting point

The infrastructure conversation is seductive. More compute, faster processing, bigger clusters — these feel like progress. And they are progress, in the right context.

When you have a validated business case, a believable ROI, signal-rich data and a well-framed modelling problem, the right infrastructure genuinely accelerates outcomes. But infrastructure applied to unvalidated data doesn’t solve the problem. It scales it.

A model trained on the wrong data, running on the best hardware available, will produce wrong answers faster and at greater cost than anyone planned for. The servers don’t know the business case is weak. The GPUs don’t know the data is empty of signal. They will process bad assumptions with perfect efficiency.

That is why the sequence matters.

Business use case → Predicted ROI → Data strategy → Data engineering and EDA → Feature validation and model readiness → Infrastructure investment

Getting that sequence right is the difference between an AI investment that delivers and one that quietly disappoints.

The IT Director with 250TB has storage. What he needs first is a conversation about what’s in it, whether it’s been tested, whether it contains usable signal, and whether it can answer the questions the business is asking. That is the conversation worth having before the servers arrive.

Closing thought

There is a version of the AI hype cycle that ends badly — and it ends badly in a specific way. Not with dramatic failure, but with quiet disappointment. Models that don’t perform. Investments that don’t deliver. Data scientists hired to build things the data was never capable of supporting.

The organisations that avoid that outcome are the ones that did the unglamorous work first. They validated the use case. They estimated the ROI. They looked at the data before they bought the infrastructure. They ran EDA before they committed to the model. They asked hard questions before they made bold commitments.

The emperor’s new clothes are always convincing until someone asks the uncomfortable question. In AI, that question is usually the same:

Have you actually tested the data?

Data readiness has to be built in, not bolted on.

Posted on

From Programming by Rules to Learning form Data

For most of software’s history, the intelligence was in the code. Today, it’s in the data. That shift changes everything — especially what you need to invest in.

Infographic comparing traditional programming and machine learning. On the left, traditional programming is depicted with a flowchart showing 'if/then' statements leading to a predetermined result. On the right, machine learning is illustrated with a data funnel leading to a network discovering patterns, highlighting the shift in the programmer's role from coding to curating data.

I had one of those shower moments this morning. You know the ones — your brain wanders somewhere unexpected and suddenly you’re solving a problem you weren’t trying to solve.

I was thinking about the time I taught my son to code a robot we’d built together. The code was beautifully simple. Turn left. Move three steps. Turn right. If you hit a wall, stop. Pure logic. Pure rules. We wrote the instructions, the robot followed them, and when it did something wrong we went back and fixed the rule.

And then I thought: when I’m a grandad — no rush — and I’m sitting down with a grandchild to do the same thing, the conversation is going to look completely different. We won’t be writing turn-left-move-three-steps rules. We’ll be feeding the robot data. We’ll be talking about what it sees, the patterns it learns from, how it gets better not because we updated the instructions but because we gave it more examples to learn from. Computer vision. Convolutional neural networks. A robot that figures out the world rather than following a script we wrote for it.

Same robot. Completely different philosophy. And somewhere between teaching my son and the future grandkids, software itself made that same journey.

For most of computing’s history, we built systems by encoding our understanding of the world directly into logic. If this, then that. If the balance is below zero, deny the transaction. If the email contains “free money” and ten exclamation marks, mark it as spam. Engineers wrote the rules, shipped the code, and the system behaved exactly as specified. The intelligence lived in the logic.

That model hasn’t disappeared — but in the domains that matter most today, it’s no longer the whole story. It’s all about data now. Patterns in the data. And understanding that shift changes how you think about what you need to invest in.

“Software will eat the world,” Marc Andreessen told us in 2011. He was right. What he didn’t mention was that software itself would eventually be powered by data. Follow that thought to its conclusion and the most important infrastructure in your organisation isn’t your application stack. It’s your data platform. And data is the power source.

When Rules Stop Being Enough

Rules-based systems are genuinely good at what they do. They’re predictable. They’re auditable. If something goes wrong, you can usually point at the line of code that caused it. For stable, well-understood processes — tax calculations, eligibility checks, simple approvals — they’re entirely fit for purpose.

The trouble starts when the problem gets messy.

Take fraud detection. You start sensibly: flag transactions above a certain amount from high-risk locations. Block IPs on a denylist. Limit transactions per minute. Clean, logical, explainable.

Then the fraudsters adapt. New attack vectors. New geographies. New patterns you didn’t anticipate. So you add more rules. Then exceptions to those rules. Then special handling for VIPs. Then manual overrides for partners. Before long, you’ve got thousands of conditions, constant firefighting, and a system that’s simultaneously brittle and impossible to fully understand — despite being built entirely from logic you wrote yourself.

At some point you hit rule sprawl, and it doesn’t end well.

The Shift: From Code as Truth to Data as Truth

Machine learning doesn’t try to specify the decision logic. Instead, it learns it — directly from examples.

Feed a model enough confirmed fraud cases alongside confirmed legitimate transactions. Give it the signals: transaction history, device fingerprint, location patterns, time of day, merchant data. Let it find the patterns. Then, when the fraud landscape shifts, you don’t sit down and rewrite hundreds of rules. You gather new examples, update the signals, retrain, and redeploy.

This is a fundamental inversion of where the intelligence lives:

  • In a rules-based system, code is the truth and data is just something to test against it.
  • In a machine learning system, data is the truth and code is the plumbing that carries it.

The code still matters enormously — it defines how data flows, how features are built, how models are trained and served. But the behaviour you see in production is now overwhelmingly a function of which data you chose, how you cleaned and joined it, how frequently you refresh it, and how well you’ve engineered the signals from it.

The same model architecture, trained on different data, can behave like a completely different product.

Output Is a Function of Input

This is the point that gets lost in conversations about AI.

Organisations invest heavily in models. They debate architectures, benchmark performance, evaluate vendors. All of that matters. But if the data flowing into those models is incomplete, inconsistent, biased or stale, no amount of model sophistication will save you.

As I covered in Garbage In, Expensive Garbage Out, the dangerous thing about modern AI isn’t that it fails obviously when the data is bad. It’s that it doesn’t. It learns whatever patterns you give it, optimises confidently for whatever labels you’ve defined, and delivers outputs at scale — even when those outputs are wrong.

The refinery metaphor runs true here. You can have the most sophisticated downstream process in the world. If contaminated feedstock is getting through the early stages, it doesn’t matter how good the refining is — what comes out the other end is still wrong. Processed wrong. Delivered at scale, with complete confidence, in entirely the wrong direction.

Output is a direct function of input. That’s not a caveat. It’s the whole game.

Data Engineering: The Refinery That Makes AI Possible

This is why data engineering has moved from the back office to the front line.

When your AI systems run on data rather than rules, the infrastructure that produces, transforms, governs and delivers that data isn’t supporting the product. It is the product — or at least, it’s what makes the product possible.

Think back to the refinery. Raw crude oil has no value in your car’s engine. It needs to go through a series of deliberate transformation stages — each one removing impurities, each one producing something more usable — before it becomes fuel you can rely on. Data works the same way.

Raw operational data, logs, clickstreams, sensor readings — these are the crude oil. Valuable in potential, useless in practice. To become model-ready, they need to flow through robust pipelines: ingested reliably, cleaned and validated, standardised across sources, transformed into features that actually capture signal, and governed throughout so you know what you have, where it came from, and whether it can be trusted.

That’s the job of the data engineer. And in a world where AI output depends on data input, that job sits at the heart of everything.

A few things have to be true for the refinery to work:

The pipelines have to be reliable. Ingestion from operational systems, logs, events and sensors. Batch and streaming paths where appropriate. Resilience to schema changes, late events and upstream failures. Without this, models starve, drift, or silently degrade on stale inputs.

The data has to be properly modelled. Standardised schemas and clear contracts between the systems that produce data and the teams that consume it. Deduplication, validation and anomaly detection built into the pipeline, not bolted on as an afterthought. Consistent definitions of what “customer” means, what “churn” means, what “conversion” means — because if those definitions vary across systems, your model is quietly learning the noise between them.

Features need to be treated as first-class assets. The signals you engineer from raw data — the features a model actually learns from — should be reusable, versioned and governed. Computed consistently whether you’re training offline or serving in real time. Not scattered across one-off notebook scripts that no one else can maintain.

Governance can’t be an afterthought. As AI moves closer to consequential decisions — credit, healthcare, hiring, public sector — knowing which data fed which model, who had access to it, and whether it was fit for that purpose stops being a compliance tick-box and becomes part of the safety story.

The loop has to close. How you capture feedback from production — user interactions, implicit signals, explicit labels — and turn it into the next generation of training data is where the compounding advantage comes from. The refinery doesn’t run once. It runs continuously.

Generative AI Turns the Dial Up, Not Off

It’s tempting to think that large language models and generative AI change this equation — that you can just point a capable model at your questions and bypass the data engineering work.

The opposite is true.

Behind every enterprise generative AI application that actually works, there are pipelines fetching the right context from your knowledge bases and data warehouses in real time. There are curated fine-tuning datasets steering the model toward the behaviour you actually want. There are feedback loops turning user interactions into better training data over time. There is, in short, a refinery — just with a different interface at the end of it.

For enterprise use cases, the differentiator is rarely the base model. It’s the quality of the data you connect it to, the rigour of the retrieval and ranking pipelines behind it, and the discipline of the data engineering that makes all of that reliable.

The plumbing is still the point.

If Data Is the Engine, Build the Right Infrastructure

The organisations that are winning with AI aren’t simply the ones with the biggest models. They’re the ones who treat data engineering as a first-class product capability — where data engineers and platform architects are in the room from the start, not brought in to implement decisions that have already been made.

They invest early in shared platform infrastructure: data lakes and warehouses, feature stores, catalogues, quality monitoring, governance and observability. Not one-off pipelines per project, but a proper refinery that serves the whole organisation.

And they build on foundations that can handle the scale and complexity of real enterprise data estates — structured tables alongside documents, images, logs and sensor data; on-premises alongside cloud and edge; batch pipelines alongside real-time streams.

That’s exactly what the Dell AI Data Platform is designed to support: a unified, modular foundation for storing, processing, governing and serving the data that modern AI workloads depend on — so data engineers can focus on building the refinery, rather than firefighting the infrastructure it sits on.

The Refinery Has to Work

The shift from rules-based systems to data-driven AI didn’t just give us more powerful software. It changed where the intelligence lives — and with it, what we need to invest in to make that software trustworthy.

When code was the truth, the bottleneck was engineers writing rules. When data is the truth, the bottleneck is the infrastructure that produces, refines, governs and delivers that data.

The refinery has to work. The pipelines have to be reliable. The fuel has to be clean. Everything downstream — every model, every decision, every output — depends on it.

And if you want to understand what happens when the refinery fails, that’s a story worth reading too.

Posted on

Garbage In, Expensive Garbage Out: Why Dirty Data Breaks Your AI

Diagram illustrating a data quality cleansing and governance engine with input elements like contaminated feedstock, bias, and outliers, leading to AI-ready data and dependable outcomes.

We’ve all heard “garbage in, garbage out.” With AI, it’s worse: garbage in, expensive garbage out.

In earlier posts in this series, we described data as the crude oil of the modern enterprise — raw, unrefined, and only valuable once it’s been properly processed. We walked through the refinery: the pipelines, the engineers, the transformation stages that turn raw data into something a model can actually learn from.

This post is about what happens when the refinery fails. When contaminated feedstock gets through. When the fuel that reaches your AI engine is dirty — and the engine runs anyway, at scale, with complete confidence, in entirely the wrong direction.

Because that’s the thing about modern AI. It doesn’t grind to a halt when the data is bad. It doesn’t throw a warning light. It just learns whatever patterns you give it, optimises for whatever labels you’ve defined, and delivers outputs with impressive precision — even when those outputs are wrong.

Bad data doesn’t give you cheap automation. It gives you expensive, automated mistakes.

What “Data Cleansing” Really Means

“Data cleansing” sounds like housekeeping. Tidy up a few rows, fix a column type, deduplicate some records. Job done.

In practice, it’s something more consequential than that. Every cleansing decision is a decision about what version of reality your AI will learn.

Back at the refinery, crude oil goes through a series of processes before it becomes usable fuel — each stage removing a different class of impurity, each stage making deliberate choices about what to keep and what to discard. Data cleansing works the same way.

At a minimum it involves fixing obvious errors: swapped fields, broken timestamps, invalid encodings. It means standardising how the same thing is represented — UK and U.K. and United Kingdom are the same country, but a model seeing them as three separate values will treat them as three separate signals. It means aligning schemas and units across systems so that a model can learn genuine patterns rather than format quirks. And it means making careful decisions about which attributes actually drive the outcome you care about, and which ones just add noise.

Here’s what makes this harder than it looks: many of those decisions involve assumptions that are invisible once they’re baked in. Which rows are outliers — and which are rare but important edge cases the business genuinely needs to handle? Which labels represent ground truth — and which carry the fingerprints of historical bias? Which data sources are trustworthy enough to be treated as signal?

In traditional reporting, a bad assumption might skew a dashboard. In AI, it gets learned, encoded, and scaled out. The model will faithfully reconstruct and amplify whatever patterns you present as truth.

How Dirty Data Shows Up

Dirty data is rarely one dramatic error. It’s a collection of small issues that quietly pull the model off course — individually manageable, collectively damaging. Here are the usual suspects.

Biased data is perhaps the most insidious. It occurs when the data doesn’t represent the population or behaviour you’re actually trying to serve. Historical decisions — approvals, pricing, hiring — get recorded as objective truth, even when they weren’t. Certain geographies, demographics or product lines are over-represented. Training sets include only “successful” cases and ignore failures or edge cases. The model faithfully learns yesterday’s prejudices and scales them out. Performance looks excellent on the same skewed distribution; it falls apart the moment you try to broaden usage.

Outliers and anomalies are trickier, because they’re not always wrong. One-off promotions, emergency discounts, crisis events, logging glitches, batch replays — these all produce data points that sit outside the normal pattern. Handle them badly and models overfit to noise. Ignore them entirely and you may have scrubbed away exactly the edge cases the business needs to handle — the fraud signals, the safety events, the compliance-relevant exceptions.

Dubious or unverified sources are a growing problem as data estates expand. Scraped web data with no curation. Third-party feeds glued into production pipelines with minimal validation. Synthetic data mixed with real data and never labelled as such. The model learns correlations that only exist in that one noisy dataset, inherits someone else’s labelling mistakes as ground truth, and produces outputs that look confident and are quietly wrong.

Nulls, blanks and missingness look simple but aren’t. A missing value in one column might mean the data was never collected, or the user chose not to answer, or the field was redacted for legal reasons, or a legacy system simply didn’t capture it. Treating all of those the same — filling everything with zero or a global mean — creates fake patterns. Models confuse “unknown” with “none”, and segments with more missing data become systematically under or over-served as a result.

Inaccurate or stale data is the fuel that’s gone off. Records without proper timestamps or versioning. Manual entry errors and mis-keyed IDs. Reference data that was accurate two years ago and hasn’t been updated since. Models trained on stale data optimise for a world that no longer exists — old pricing, old customer behaviour, old market conditions — and deliver what look like strong offline metrics that evaporate in production.

Inconsistent definitions are often the hardest to catch because the data looks clean. “Customer” means the billing account in one system, the individual in another, the household in a third. “Churn” means no login for thirty days in one team’s definition, no revenue in ninety days in another’s. The model trains on a Frankenstein label that no one fully agrees on, produces outputs with impressive AUC scores, and is quietly solving the wrong problem.

Why AI Gets It Wrong Even When the Model Looks Fine

Most AI failures aren’t about the algorithm. They’re about the data pipeline.

Dirty data corrupts the learning process in ways that don’t show up until the model is in production. If your labels are biased or conceptually wrong, your model is optimising for the wrong behaviour — and it will do so very efficiently. If the training and validation data share the same quirks (the same logging artefacts, the same process shortcuts, the same dummy values), the model will look excellent right up until it meets real-world data that doesn’t carry those quirks.

Modern models are extraordinarily good at interpolating through messy data. That’s part of what makes them powerful. It’s also what makes dirty data so dangerous. You can get good-looking metrics on a flawed test set, a slick demo that works beautifully on carefully prepared examples, and a deployment that quietly fails in the wild. The model isn’t broken. It’s just running on contaminated fuel — confidently, at scale, in the wrong direction.

Timing Is Part of Data Quality

We tend to talk about data quality as if it were purely about correctness. In AI systems, freshness is just as important.

If you’re making real-time decisions on yesterday’s snapshot, your fraud models are lagging the attackers, your recommendation models are pushing last week’s interests, and your operational models are optimising for a backlog that’s already changed.

But the deeper problem is concept drift. The world changes — new products, new pricing, new channels, macro events that reshape customer behaviour entirely. If your training data doesn’t keep up, the model keeps extrapolating from patterns that no longer hold, with no way to know that the ground has shifted beneath it.

And then there are feedback loops. Many AI systems are closed loops: the model makes decisions, those decisions influence what data gets collected next, and that data is used to retrain the model. If the early data is dirty or biased, the loop amplifies the problem. The model stops seeing counter-examples. Segments that are mis-served stay mis-served. The system drifts quietly into a self-reinforcing corner of the feature space.

Data quality, properly understood, is correctness multiplied by relevance multiplied by timeliness. If any of those three factors drops to zero, the output is still expensive garbage — however impressive the model architecture.

Smarter Cleansing, Not Aggressive Pruning

Given all of this, it’s tempting to treat cleansing as “delete anything that looks messy.” That’s the wrong instinct — and it can be as damaging as doing nothing.

You don’t want to scrub away minority groups because they represent small sample sizes. You don’t want to discard edge cases that reveal failure modes you need to handle. You don’t want to erase records of bad historical outcomes that show you exactly where bias or process issues lived.

Smarter cleansing means diagnosing and documenting bias rather than pretending it isn’t there — quantifying where the data is skewed, and deciding consciously whether to re-weight, augment or treat those segments separately. It means treating outliers as signals rather than inconveniences, and recognising that for some use cases — fraud, anomaly detection, safety — the outliers are the whole point. It means making missingness explicit, using indicator variables or separate categories rather than silently filling gaps with a magic constant. And it means tracking lineage and provenance so that when something goes wrong in production, you can debug the pipeline — not the model weights.

Cleansing isn’t housekeeping. It’s data governance. And it’s the most important thing you can do for AI that’s actually trustworthy.

Keeping the Refinery Running: Where Dell Comes In

All of this is straightforward in principle and genuinely hard in practice — especially when your data estate is spread across on-premises infrastructure, cloud environments and edge locations, and you’re working with structured tables alongside documents, images, logs and sensor data simultaneously.

The challenge isn’t understanding what clean data looks like. It’s building the refinery infrastructure that produces it consistently, at scale, without burying your data engineers in manual work.

That’s the problem Dell is focused on solving.

It starts before the pipelines. Dell’s Data Strategy Services work with data engineers, architects and business stakeholders to define what “good data” actually means for each AI use case — not generically, but specifically: which sources are trustworthy, where bias and gaps exist, what quality and governance requirements each workload demands, and what the end-to-end data management roadmap needs to look like to get there. Engineers build better pipelines when they have a clear, agreed target for the fuel they’re supposed to be producing.

The foundation underneath those pipelines is the Dell AI Data Platform — a unified, modular platform that stores and manages structured, semi-structured and unstructured data through storage engines like PowerScale and ObjectScale, and uses Data Engines to clean, organise and optimise information from applications, devices and other sources. Built-in security and governance — access controls, data masking, encryption and cyber-resilient features — mean that sensitive data stays protected without creating friction for the teams who need to work with it.

On top of that foundation, a significant portion of what data engineers spend their time on — deduplication, format standardisation, missing data imputation, anonymisation — can be automated. Dell’s platform combines AI-driven preprocessing with automated cleansing at scale, so engineers aren’t manually patching pipelines for every new data source. The result is clean, consistently structured data flowing through to models, rather than a patchwork of workarounds that looks fine until it doesn’t.

But clean storage and cleansing aren’t enough on their own. AI isn’t a single batch job — it’s a continuous lifecycle: data preparation, validation, model training, deployment, monitoring, retraining, and back again. The Dell Data Orchestration Engine acts as the control plane for that entire loop, connecting ingestion, pipeline automation, validation, model triggers and governance enforcement across structured and unstructured, batch and streaming data. Critically, it enforces data quality gates before models ever see the data — so bias, outliers and missingness don’t silently migrate into production systems while no one is watching.

And wrapping all of it is the expertise to make it work in practice. Dell’s data engineering services help teams design and implement pipelines that tag, cleanse, label and anonymise data to produce AI-ready datasets in the right format at the right time — all as part of the broader Dell AI Factory with NVIDIA, a vertically integrated stack of storage, data engines, orchestration, GPUs and services built for the next decade of AI workloads.

Expensive Models, or Dependable Outcomes?

Powerful models make it easy to forget that data is the real engine.

You can invest heavily in GPUs, premium model architectures and sophisticated orchestration. But if the fuel going into those systems is dirty — late, inconsistent, biased, poorly understood — you won’t get cheap insight. You’ll get expensive, high-confidence mistakes, delivered at scale, by a system that has no idea anything is wrong.

The organisations that win with AI won’t simply have the biggest models. They’ll have the cleanest, best-understood data, flowing through governed and automated pipelines, built by data engineers who are spending their time on value rather than firefighting.

That’s the shift the Dell AI Data Platform is designed to support: from garbage in, expensive garbage out — to clean data in, dependable outcomes out. The refinery has to work. Everything downstream depends on it.

If you’d like to explore this further, Dell has gone deeper on both the data-quality story and the platform behind it:
– Keep Your AI Engine Running with Good, Clean Data
– Dell AI Data Platform

Posted on

How Dell AI and Data Platform (AIDP) Helps Charlie and Clare Deliver Real Outcomes

So far in this series we’ve built up a picture of the data journey from the ground up — from raw events to refined data, from pipelines to insight, from Charlie’s world to Clare’s. In this post it’s time to connect that picture to the technology designed to support it.

The platform in question is Dell’s AI and Data Platform — AIDP. But before diving into what it does, it’s worth being clear about how to measure whether any platform is actually working.

The value of a data and AI platform isn’t measured in terabytes or TFLOPS. It’s measured in how fast you can go from idea to insight to action, how confident you are in the data and models behind key decisions, and how many AI and analytics initiatives actually reach production rather than dying somewhere between the notebook and the business.

That’s the lens to apply here.

What Dell AI and Data Platform actually is

Think of AIDP not as a single product but as an integrated foundation — Dell’s storage and compute brought together with the software and tools needed for modern data and AI workloads. It’s designed to support the full journey: data engineering, analytics, and AI and ML, end to end.

On top of Dell’s infrastructure, AIDP incorporates modern data platform patterns — data warehouse, data lake, and data lakehouse — a unified query layer via Starburst to access data wherever it lives, AI and ML tooling for model development and operations, and governance and observability across the stack. In other words, it’s an opinionated foundation for the world Charlie and Clare actually work in every day.

How AIDP helps Charlie

Charlie’s biggest frustrations are predictable: platforms that can’t handle volume without drama, bespoke pipelines that multiply every time a new team needs data, and blind spots in quality and observability that turn into production incidents.

AIDP addresses all three. The Dell-validated infrastructure gives Charlie a solid, scalable backbone for both batch and streaming workloads — predictable performance, consistent environments for orchestration and monitoring, and the ability to onboard new data sources without reopening architecture debates every time. Charlie can design against it with confidence rather than working around it.

On the data platform side, AIDP supports warehouse, lake, and lakehouse patterns on the same infrastructure, so Charlie isn’t forced into a single storage model. Governed structured analytics can live alongside flexible data science workloads, on the same foundation, without creating yet another silo. And the integration with Starburst as a unified query layer means Charlie can connect existing sources once, define logical schemas and curated views centrally, and expose them to multiple consumers — BI tools, notebooks, AI platforms — via standard SQL. ETL and ELT still matter for refining critical datasets, but Charlie now gets to choose when to materialise data and when to virtualise and query in place. Less time writing and fixing bespoke pipelines. More time building reusable, governed data products.

Finally, AIDP gives Charlie integration points for DataOps tooling — testing, CI/CD, data quality validation, centralised security and governance controls. Fewer blind spots. Fewer “we can’t trust these numbers” conversations. And a better compliance posture as a baseline rather than an afterthought.

How AIDP helps Clare

Clare’s frustrations are equally predictable: too much time hunting and reconciling data, queries and training runs that struggle on realistic dataset sizes, and models that work in development but never make it to production.

The same curated views and data products Charlie defines in AIDP are exposed directly to Clare’s tooling — BI platforms for analysts, notebooks and ML frameworks for data scientists — with governance and security enforced at the platform level rather than left to ad-hoc exports. Metric definitions are consistent. Lineage is traceable. Clare spends less time cleaning and reconciling, and more time actually analysing and modelling.

On the compute side, AIDP is designed to handle both analytical and AI workloads — scalable compute for large joins and aggregations, GPU and CPU support for model training and inference, storage tuned for high-throughput parallel access. Clare can run complex queries and train models on realistic data, not tiny samples that fit on a laptop.

And for data scientists specifically, the path from notebook to production is where AIDP earns its keep most clearly. A consistent environment across development, test, and production. Integration with MLOps tools for deployment and monitoring. Charlie’s data pipelines and Clare’s model workflows running on the same underlying platform. The result is fewer science projects and more AI features actually embedded in products and processes.

The bigger picture

Step back from the personas for a moment and the business case is straightforward. When Charlie and Clare are well supported, organisations make faster decisions on trusted data, data talent spends more time on high-value work and less on plumbing, and AI projects reach production instead of stalling between experiment and deployment.

But perhaps the most important thing AIDP offers is a platform that grows with your ambitions. Most organisations are on a journey — from basic reporting, to advanced analytics, to machine learning, to GenAI and AI-enhanced applications. That journey rarely happens in a straight line, and it rarely happens all at once. AIDP is designed as a foundation that supports each step without requiring a rip-and-replace every time the ambition moves on. Start with strong data engineering and analytics. Add predictive models when the data foundations are ready. Layer on GenAI when governance and quality can support it.

Going back to the analogy that’s run through this series: AIDP is the refinery infrastructure — the tanks, the pipes, the processing units — that makes it possible to keep producing reliable data fuel as demand grows and the grades required become more sophisticated.

In the next post, we’ll go deeper into the architecture of AIDP itself — how the pieces fit together technically, and how that maps back to the data flows and personas we’ve been building throughout this series.

Posted on

Meet Clare, the Data Analyst and Data Scientist

In the last post we met Charlie, the Data Engineer — the person who designs and runs the pipelines that move data from raw to reliable. But pipelines aren’t the destination. They’re the supply chain. The destination is decisions.

That’s where Clare comes in.

Depending on the organisation, Clare might carry the title of Data Analyst, BI Analyst, Data Scientist, or some hybrid in between. Titles vary, but the core idea is the same: if Charlie is responsible for getting the data ready, Clare is responsible for making it mean something.

Why Clare matters

It’s tempting to think of a data platform as an infrastructure problem — storage, compute, pipelines. Get those right and the job is done. But the business doesn’t care about petabytes or processing cores. It cares about answers:

  • Why did churn spike last quarter?
  • Which products are actually profitable by customer segment?
  • Where are we wasting marketing spend?
  • What early signals predict a failure or a fraud event?

Clare is the person who sits between the data estate and the decisions that leaders actually make. Understanding what Clare needs — and how a platform either helps or hinders them — is just as important as understanding Charlie.

Clare as a Data Analyst: questions, context, and communication

In an analyst role, Clare’s job starts with a business question — “why are returns higher in this region?” or “where are we losing customers in the journey?” — and works backwards into the data. Which tables matter? How do we define the metric consistently? What filters and segments do we need?

That requires a good understanding of the business domain, a working knowledge of the data model Charlie has built, and enough SQL and data skills to pull and join the right datasets. If the underlying platform is inconsistent, slow, or poorly documented, Clare spends more time hunting for data than answering questions — and the business waits longer for insight. And we know there’s cost and lost opportunity when waiting.

Once she has the data, Clare explores, visualises, and explains. The dashboards and reports are what most people see. But the real value is in the interpretation — which patterns are robust and which are noise, what changed and when, and where the data is strong or weak. Good analysts don’t just produce charts. They tell the story of the data in a way business leaders can act on.

Over time, that work compounds. Ad-hoc analysis becomes reusable dashboards. Useful logic gets worked back with Charlie into shared views and models. Agreed metric definitions — one version of “churn”, “active user”, “profitability” — slowly replace the spreadsheet chaos that plagues most organisations. The loop closes: business questions feed data exploration, which feeds agreed definitions, which feed shared data products.

Clare as a Data Scientist: from patterns to models

In a data scientist role, Clare goes a step further. Where the analyst might conclude that customers with behaviour X are more likely to churn, the scientist trains a model to predict it — validates it on historical data, packages it, and works towards deploying it somewhere it can actually be used.

The same applies to demand forecasting, fraud detection, recommendation engines, or predictive maintenance. The goal is to turn data and hypotheses into models that can reliably predict or classify future events, and then get those models out of notebooks and into production.

That last part is where the platform really shows its hand. A model isn’t valuable until it runs reliably, receives fresh data regularly, and is monitored for drift. Getting from proof of concept to production requires Clare to work closely with Charlie and ML engineering teams — and it requires a platform that is consistent across development, test, and production environments. When that foundation is solid, the journey from notebook to deployed model is achievable. When it isn’t, models die in notebooks.

What Clare needs from the platform

Whether she’s in analyst or scientist mode, Clare’s needs come down to five things:

Access — the ability to get to the data she needs without raising a ticket for every new question. Trust — numbers that are consistent across tools and teams, with lineage she can follow back to source. Performance — queries and model training runs that complete in minutes, not hours. Flexibility — the freedom to use the tools she’s comfortable with, whether that’s SQL, Python, R, or a BI front end, all drawing from the same curated data layer. And stability — a platform that behaves consistently when new workloads appear, rather than one that needs babysitting.

None of those are unreasonable asks. But they all trace back to decisions made much further down the stack — in the data pipelines Charlie operates, in the platform architecture underneath, and in the governance and documentation that surrounds it all.

Charlie and Clare: the same chain

Put the two personas together and a simple picture emerges:

Charlie designs and operates the pipelines. He turns raw data into reliable, documented, timely datasets — the refined fuel. Clare takes that fuel and turns it into insight, models, and decisions — the journeys the business actually needs to take.

Everything else — storage, compute, orchestration, infrastructure — exists to help Charlie and Clare do those jobs well. If they can’t get the data they need, trust the numbers, or iterate quickly on new questions and models, the sophistication of what sits underneath doesn’t really matter.

In the next post, we’ll start connecting these personas and data flows to the architecture of a modern data platform — and show how the layers fit together to support the full journey from raw data to decision.

Posted on

Meet Charlie, the Data Engineer

Before we look at what a modern data platform actually looks like, it’s worth pausing to ask: who is it built for?

Platforms don’t create value on their own. People do. And in a world that runs on data and AI, there’s a set of roles that sit at the heart of that value chain — people who design, build, and consume the pipelines that turn raw data into decisions. Understanding who they are, and what they actually need, changes how you think about everything else.

The first person worth introducing is Charlie — the Data Engineer.

Why the Data Engineer exists

Not long ago, data lived inside systems. The database behind the ERP. The CRM. The billing platform. IT looked after servers, storage and availability. Business teams raised tickets when they wanted a report. That model worked when data was a by-product of running the business.

It doesn’t work like that any more.

When analytics needs to be near real-time rather than batched once a month, when AI teams need large consistent training sets and continuous feature feeds, and when data scientists are expected to build models that drive real decisions — someone has to sit between raw application data and the people and systems consuming it.

That’s Charlie.

What Charlie actually does

At its core, Charlie’s job is to move data reliably through its lifecycle — from raw and messy, to clean, modelled, and ready for use. If a dataset appears in a dashboard, a model, or an AI assistant, somewhere behind it is a data engineer making it flow.

Think back to the refinery analogy. If raw data is crude oil, Charlie is the refinery engineer — designing and operating the pipelines, monitoring the process, and making sure the right grade of fuel reaches the right engine at the right time.

In practice that means four things.

First, building and operating pipelines that handle ingestion, transformation, storage, and serving of data — repeatably, reliably, and at production grade. Not one-off scripts that work once and break on a Monday morning.

Second, turning messy source data into trustworthy data products — clean, modelled, documented, and timely. Tables and views that reflect how the business actually thinks: customers, products, assets, events. Data as a product, not just a dump from a system.

Third, making pragmatic technology choices. It’s easy to chase the latest tool or architectural pattern. Charlie doesn’t have that luxury. Every decision — warehouse, lake, lakehouse, stream processor, orchestration engine — has to be weighed against cost, performance, operational reality, and whether it will actually play nicely with the rest of the stack.

Fourth, making governance real. Data quality isn’t a policy document; it lives in the pipelines Charlie builds and operates. Validation checks, lineage tracking, access controls, schema versioning — this is where the business’s aspiration for a “single source of truth” either becomes reality or stays a slide on a deck.

And through all of it, Charlie collaborates — with architects who set direction, analysts who know what business users need, scientists who know what their models require, and stakeholders who own the outcomes. When that collaboration works, projects move from proof of concept to production. When it doesn’t, you get dashboards nobody trusts and models that never leave the lab.

Why this matters

Understanding Charlie’s world reframes what a data platform needs to be. It’s not just storage and compute. It’s the foundation that keeps pipelines reliable, helps teams serve analysts and scientists faster, and makes a complex job simpler rather than more complicated.

In the next post, we’ll meet the colleagues downstream of Charlie — the data analyst and the data scientist — and see how all three fit into the same data value chain we’ve been building through this series.

Posted on

Just What is the Fuss About Data?

We talk about data constantly, but most of us never stop to ask what it actually is. In everyday life, we swim in data without noticing. None of it feels dramatic — but all of it is data:

  • Your phone counting steps and heart rate during a walk
  • Your car logging speed, mileage, battery or fuel usage
  • A supermarket barcode scanner recording what you bought and when
  • A streaming app tracking what you watch, pause, rewind, and abandon
  • A website logging every click, scroll, and form field you touch

Each of those is simply a captured observation about an event, a behaviour, or a state. On its own, it’s just a record. Raw. Inert. Waiting.

Data only becomes information when it has context. And it only becomes knowledge when it drives a decision. That three-step ladder — data, information, knowledge — is worth holding onto, because it explains why so many organisations sit on vast amounts of data and still struggle to act on it.

Most people only ever see the polished outputs: an activity ring closing on a smartwatch, a “recommended for you” row on Netflix, a monthly bank statement. They never see the raw data stream behind it.

Here’s a useful way to think about what separates those who do.

In The Matrix, most people only see the illusion of the world. A few stare directly at the green code and can see “the woman in the red dress” — the meaningful signal hidden inside the data stream. Modern businesses face exactly the same divide:

  • Some only see systems and reports on the surface.
  • Others have learned to read the data behind those systems — and spot patterns, risks, opportunities, and inefficiencies that everyone else misses.

Data is the stream. The real advantage is knowing how to find the woman in the red dress. How to find the signals in the noise.

Why Data Matters to Modern Business and Decision-Making

If data only becomes knowledge when we can interpret and act on it, then modern business success is really a competition in how well you can move from raw events → information → knowledge → decisions → actions.

Almost every important decision in a business can be traced back to data, even if people don’t realise it:

  • Which products do we build next? Driven by sales, returns, customer feedback, and competitor moves.
  • Where do we invest, and where do we cut back? Driven by margins, pipeline, utilisation, risk, and market signals.
  • How do we improve customer experience? Driven by NPS, churn, ticket volumes, digital journeys, and operational metrics.
  • How do we run our operations? Driven by inventory levels, lead times, failure rates, staffing and capacity data.

In the past, a lot of this ran on experience and instinct, with data used mainly to justify decisions after the fact. Today, that simply doesn’t scale. Markets move faster, supply chains are more fragile, customers have more choice and more voice, and regulators expect more transparency. You can’t manage that complexity with periodic spreadsheets and static reports. You need continuous, trustworthy, integrated data feeding analytics and — increasingly — AI.

That’s where the Matrix metaphor earns its keep. Two companies can have access to roughly the same raw data:

  • One treats it as exhaust — something systems happen to log.
  • The other treats it as knowledge fuel — the foundation every decision is built on.

The second company invests in building what you might think of as a supply chain for data: capturing it consistently from key systems and devices, bringing it together so it can be compared and combined, cleaning it so people actually trust it, and modelling it so it reflects how the business really works. Only then do analytics and AI have something meaningful to work with.

Because here’s the uncomfortable truth:

The quality of your decisions is limited by the quality of the data that reaches your decision-makers — not by how clever your dashboards or algorithms are.

That brings us to the next question: how does raw data from all these sources actually become decision-ready? The answer lies in the layers and pipelines that sit between source systems and the dashboards or AI models that consume them — and that’s where ideas like ETL and ELT come in.

From Raw Oil to Refined Fuel: Why Data Must Be Transformed (refined)

So far we’ve established that raw data needs work before it becomes useful. But it’s worth being honest about just how much work that actually is.

Think of crude oil straight out of the ground. It’s thick, dirty, and full of impurities. Enormously valuable — but you can’t pour it directly into a car engine or a jet turbine. It has to go through a refinery first, to become petrol, diesel, or kerosene. Only then can engines use it efficiently and safely. Data is exactly the same.

The realities of raw data

Raw data coming off real systems is messy in predictable ways. Outliers — impossible values like negative ages or a million units sold in a single second — creep in through bugs or misuse. Fields are blank, records are partial, events are logged without IDs or timestamps. Rows get truncated, formats clash, codes are inconsistently typed. And then there’s the sheer volume of data that simply isn’t relevant — verbose logs, granular events, fields nobody needs for the use case at hand.

Feed that raw stream directly into analytics or AI and you get dashboards people don’t trust, models that learn the wrong patterns, and insights that fall apart the moment someone checks the underlying numbers. Just like engines need refined fuel, analytics and AI need refined data.

ETL and ELT: the data refineries

This is where data transformation comes in — and why concepts like ETL and ELT matter. Both approaches take raw data from source systems and turn it into something cleaner, more consistent, more focused, and easier for analytics and AI to consume. They just differ in where the heavy lifting happens.

With ETL (Extract – Transform – Load), data is pulled from source systems, transformed in a separate processing layer — cleaned, standardised, reshaped — and only then loaded into the target warehouse. Think of it as a refinery that sits outside the storage tank: crude comes in one end, refined product goes out the other. This approach made sense when storage was expensive and tightly controlled, and you wanted only highly curated data to ever land in the warehouse.

With ELT (Extract – Load – Transform), the order flips. Raw data lands in the warehouse or data lake first, and transformation happens inside the platform using its own compute engine. The refinery, in effect, is built inside the tank. This became the dominant approach as storage and compute got cheaper, and teams wanted the flexibility to experiment directly against raw data before deciding what to curate.

Why transformation matters

Regardless of which approach you use, the goals are the same: fix or flag bad data, standardise and integrate across sources, reduce to what actually matters for the use case, and shape the data into structures that reflect how the business thinks — customers, products, assets, locations, periods.

Done well, analysts spend more time analysing and less time cleaning. Dashboards match reality — and match each other. AI models train on consistent, documented data rather than whatever happened to be easiest to pull.

Which brings us back to the analogy that ties it all together:

Raw data is crude oil. ETL and ELT are your refineries. Analytics and AI are the engines. And business decisions are the journeys you can take once you’ve got reliable fuel.

In my next blog, we’ll look inside a modern data platform — exploring the layers from raw through to curated and semantic — and show exactly where this refining work happens at each stage.

Posted on

Data: The Foundation of Analytics and AI

AI is not the starting point. Analytics is. And for most organisations, getting that order wrong is the single biggest reason AI projects disappoint.

Every week I’m pulled into conversations framed as “AI strategy” that are, in reality, data and analytics problems in disguise. Beneath the ambition — copilots for sales teams, chatbots for customer service, predictive maintenance on the shop floor — you usually find the same things: siloed application data, legacy warehouses built for yesterday’s reporting, data lakes that became data swamps, and spreadsheets filling the gaps.

Until those foundations are solid, AI won’t fix the problem. It will just produce more sophisticated confusion.

This first post in the series makes that case explicitly. Before we argue warehouse vs lake vs lakehouse, we need a shared view of how analytics underpins decision-making — and how AI builds on, not replaces, that foundation.

The analytics spectrum: a ladder, not a leap

Most organisations jump straight to AI use cases while still struggling with the basics. It helps to think of analytics as a spectrum, where each stage depends on the one before it.

Descriptive – “What happened?” Reports, dashboards, KPIs. Revenue by region, tickets by channel, machines that failed by shift. This is where most business decisions are still made today, and where most organisations need the most work.

Diagnostic – “Why did it happen?” Drill-downs, cohort comparisons, root-cause analysis. Why did returns spike in that region? Why did NPS drop after the launch? This is where data moves from a record of events to an explanation of them — and many “analytics projects” live here, without any AI in sight.

Predictive – “What is likely to happen?” Now we step into data science: demand forecasting, churn prediction, time-to-failure estimates. Classical ML techniques — regression, classification, time-series models — let businesses anticipate outcomes and act before issues hit.

Prescriptive – “What should we do?” From prediction to recommended action. Given stock levels, demand signals, and lead times, what’s the optimal replenishment plan? This is optimisation, simulation, and decision engines — again, often without a single line of GenAI.

AI and GenAI – “What can we automate, augment, and generate?” Finally, AI on top of that foundation: ML models embedded in workflows, copilots that explain performance, RAG systems that answer questions over enterprise data. Powerful — but only as powerful as the analytics beneath them.

The dependency chain matters: weak descriptive and diagnostic analytics produce weak predictive models. Weak analytics produces AI that surfaces those weaknesses faster and at greater scale.

What this looks like in practice

Manufacturing: the cost of skipping the basics

A mid-sized manufacturer wanted ML-based predictive maintenance. Reasonable ambition. But when we dug in, OEE dashboards didn’t exist, downtime wasn’t being captured consistently, and maintenance history lived in three disconnected systems. There was no foundation to train a model on — and no way to validate one if you did.

The right first move wasn’t a model. It was centralising OT data from IoT sensors, PLC logs, and work orders, then building basic descriptive analytics: downtime by line, scrap rates by batch, top failure modes by site. That work alone drove measurable improvements. Statistical MTBF estimates followed. ML-based failure prediction came later — and actually worked, because the data beneath it was trusted.

The lesson isn’t that AI is bad. It’s that the sequence matters.

Retail: analytics as the backbone of personalisation

In retail and e-commerce, AI gets a lot of credit for personalisation — but the analytics layer does most of the heavy lifting. Sales trends, basket analysis, promotion effectiveness, seasonality-aware forecasting at category or store level: these are all analytically tractable problems that don’t require a foundation model to solve.

ML propensity models and recommendation engines amplify that foundation. GenAI can summarise performance and suggest campaign tweaks. But retailers who skip straight to AI without first understanding their own sales patterns, customer segments, and channel dynamics tend to automate their confusion rather than their insight.

Financial services: where auditability limits shortcuts

In heavily regulated sectors, the stakes of a weak analytics foundation are higher than embarrassment — they’re regulatory. Descriptive and diagnostic analytics aren’t optional extras here; they’re the basis for regulatory reporting, risk management, and audit trails.

ML fraud detection and GenAI investigation assistants are genuinely valuable in financial services. But they’re layered in carefully, on top of scorecards, delinquency models, and segment-level risk analysis that have been validated and governed over years. The analytics estate isn’t just fuel for AI — it’s the compliance record that makes AI defensible.

AI readiness is analytics maturity

When organisations talk about AI readiness, the conversation often turns to GPU capacity and model choice. For most businesses, that’s the wrong question. The right questions are:

  • Do you have trusted KPIs and shared metric definitions?
  • Can you trace those metrics back to source systems?
  • Are your analysts and business users comfortable challenging assumptions with data?

If the answer is no, adding AI won’t close those gaps. It will expose them at scale.

Two things worth noting for teams at this stage. First, analytics teams are the natural bridge to AI — they already understand the business context, the data quirks, and the stakeholder expectations that make models useful rather than academic. Second, the artefacts analytics teams produce — curated datasets, semantic models, metric definitions — are exactly what AI projects need: feature stores for ML, knowledge bases for RAG. Reusing them avoids rebuilding from scratch in every AI initiative.

The better the analytics foundation, the faster and safer AI can be introduced. Your data platform decision — warehouse, lake, or lakehouse — is an analytics decision first and an AI decision second. I’ll unpack those architecture choices in the next post.

Don’t start with the model. Start with the metrics.

Pick one high-impact decision your organisation makes today. Write down what data is used, how it’s analysed, and what would change if the analytics were more reliable — before AI enters the picture.

That single exercise will tell you more about your AI readiness than any consultant assessment. And it will tell you whether your next conversation should be about models, or about the metrics and data quality that would make those models worth deploying.