Page 2 of 3

From Programming by Rules to Learning form Data

For most of software’s history, the intelligence was in the code. Today, it’s in the data. That shift changes everything — especially what you need to invest in.

Infographic comparing traditional programming and machine learning. On the left, traditional programming is depicted with a flowchart showing 'if/then' statements leading to a predetermined result. On the right, machine learning is illustrated with a data funnel leading to a network discovering patterns, highlighting the shift in the programmer's role from coding to curating data.

I had one of those shower moments this morning. You know the ones — your brain wanders somewhere unexpected and suddenly you’re solving a problem you weren’t trying to solve.

I was thinking about the time I taught my son to code a robot we’d built together. The code was beautifully simple. Turn left. Move three steps. Turn right. If you hit a wall, stop. Pure logic. Pure rules. We wrote the instructions, the robot followed them, and when it did something wrong we went back and fixed the rule.

And then I thought: when I’m a grandad — no rush — and I’m sitting down with a grandchild to do the same thing, the conversation is going to look completely different. We won’t be writing turn-left-move-three-steps rules. We’ll be feeding the robot data. We’ll be talking about what it sees, the patterns it learns from, how it gets better not because we updated the instructions but because we gave it more examples to learn from. Computer vision. Convolutional neural networks. A robot that figures out the world rather than following a script we wrote for it.

Same robot. Completely different philosophy. And somewhere between teaching my son and the future grandkids, software itself made that same journey.

For most of computing’s history, we built systems by encoding our understanding of the world directly into logic. If this, then that. If the balance is below zero, deny the transaction. If the email contains “free money” and ten exclamation marks, mark it as spam. Engineers wrote the rules, shipped the code, and the system behaved exactly as specified. The intelligence lived in the logic.

That model hasn’t disappeared — but in the domains that matter most today, it’s no longer the whole story. It’s all about data now. Patterns in the data. And understanding that shift changes how you think about what you need to invest in.


“Software will eat the world,” Marc Andreessen told us in 2011. He was right. What he didn’t mention was that software itself would eventually be powered by data. Follow that thought to its conclusion and the most important infrastructure in your organisation isn’t your application stack. It’s your data platform. And data is the power source.

When Rules Stop Being Enough

Rules-based systems are genuinely good at what they do. They’re predictable. They’re auditable. If something goes wrong, you can usually point at the line of code that caused it. For stable, well-understood processes — tax calculations, eligibility checks, simple approvals — they’re entirely fit for purpose.

The trouble starts when the problem gets messy.

Take fraud detection. You start sensibly: flag transactions above a certain amount from high-risk locations. Block IPs on a denylist. Limit transactions per minute. Clean, logical, explainable.

Then the fraudsters adapt. New attack vectors. New geographies. New patterns you didn’t anticipate. So you add more rules. Then exceptions to those rules. Then special handling for VIPs. Then manual overrides for partners. Before long, you’ve got thousands of conditions, constant firefighting, and a system that’s simultaneously brittle and impossible to fully understand — despite being built entirely from logic you wrote yourself.

At some point you hit rule sprawl, and it doesn’t end well.


The Shift: From Code as Truth to Data as Truth

Machine learning doesn’t try to specify the decision logic. Instead, it learns it — directly from examples.

Feed a model enough confirmed fraud cases alongside confirmed legitimate transactions. Give it the signals: transaction history, device fingerprint, location patterns, time of day, merchant data. Let it find the patterns. Then, when the fraud landscape shifts, you don’t sit down and rewrite hundreds of rules. You gather new examples, update the signals, retrain, and redeploy.

This is a fundamental inversion of where the intelligence lives:

  • In a rules-based system, code is the truth and data is just something to test against it.
  • In a machine learning system, data is the truth and code is the plumbing that carries it.

The code still matters enormously — it defines how data flows, how features are built, how models are trained and served. But the behaviour you see in production is now overwhelmingly a function of which data you chose, how you cleaned and joined it, how frequently you refresh it, and how well you’ve engineered the signals from it.

The same model architecture, trained on different data, can behave like a completely different product.


Output Is a Function of Input

This is the point that gets lost in conversations about AI.

Organisations invest heavily in models. They debate architectures, benchmark performance, evaluate vendors. All of that matters. But if the data flowing into those models is incomplete, inconsistent, biased or stale, no amount of model sophistication will save you.

As I covered in Garbage In, Expensive Garbage Out, the dangerous thing about modern AI isn’t that it fails obviously when the data is bad. It’s that it doesn’t. It learns whatever patterns you give it, optimises confidently for whatever labels you’ve defined, and delivers outputs at scale — even when those outputs are wrong.

The refinery metaphor runs true here. You can have the most sophisticated downstream process in the world. If contaminated feedstock is getting through the early stages, it doesn’t matter how good the refining is — what comes out the other end is still wrong. Processed wrong. Delivered at scale, with complete confidence, in entirely the wrong direction.

Output is a direct function of input. That’s not a caveat. It’s the whole game.


Data Engineering: The Refinery That Makes AI Possible

This is why data engineering has moved from the back office to the front line.

When your AI systems run on data rather than rules, the infrastructure that produces, transforms, governs and delivers that data isn’t supporting the product. It is the product — or at least, it’s what makes the product possible.

Think back to the refinery. Raw crude oil has no value in your car’s engine. It needs to go through a series of deliberate transformation stages — each one removing impurities, each one producing something more usable — before it becomes fuel you can rely on. Data works the same way.

Raw operational data, logs, clickstreams, sensor readings — these are the crude oil. Valuable in potential, useless in practice. To become model-ready, they need to flow through robust pipelines: ingested reliably, cleaned and validated, standardised across sources, transformed into features that actually capture signal, and governed throughout so you know what you have, where it came from, and whether it can be trusted.

That’s the job of the data engineer. And in a world where AI output depends on data input, that job sits at the heart of everything.

A few things have to be true for the refinery to work:

The pipelines have to be reliable. Ingestion from operational systems, logs, events and sensors. Batch and streaming paths where appropriate. Resilience to schema changes, late events and upstream failures. Without this, models starve, drift, or silently degrade on stale inputs.

The data has to be properly modelled. Standardised schemas and clear contracts between the systems that produce data and the teams that consume it. Deduplication, validation and anomaly detection built into the pipeline, not bolted on as an afterthought. Consistent definitions of what “customer” means, what “churn” means, what “conversion” means — because if those definitions vary across systems, your model is quietly learning the noise between them.

Features need to be treated as first-class assets. The signals you engineer from raw data — the features a model actually learns from — should be reusable, versioned and governed. Computed consistently whether you’re training offline or serving in real time. Not scattered across one-off notebook scripts that no one else can maintain.

Governance can’t be an afterthought. As AI moves closer to consequential decisions — credit, healthcare, hiring, public sector — knowing which data fed which model, who had access to it, and whether it was fit for that purpose stops being a compliance tick-box and becomes part of the safety story.

The loop has to close. How you capture feedback from production — user interactions, implicit signals, explicit labels — and turn it into the next generation of training data is where the compounding advantage comes from. The refinery doesn’t run once. It runs continuously.


Generative AI Turns the Dial Up, Not Off

It’s tempting to think that large language models and generative AI change this equation — that you can just point a capable model at your questions and bypass the data engineering work.

The opposite is true.

Behind every enterprise generative AI application that actually works, there are pipelines fetching the right context from your knowledge bases and data warehouses in real time. There are curated fine-tuning datasets steering the model toward the behaviour you actually want. There are feedback loops turning user interactions into better training data over time. There is, in short, a refinery — just with a different interface at the end of it.

For enterprise use cases, the differentiator is rarely the base model. It’s the quality of the data you connect it to, the rigour of the retrieval and ranking pipelines behind it, and the discipline of the data engineering that makes all of that reliable.

The plumbing is still the point.


If Data Is the Engine, Build the Right Infrastructure

The organisations that are winning with AI aren’t simply the ones with the biggest models. They’re the ones who treat data engineering as a first-class product capability — where data engineers and platform architects are in the room from the start, not brought in to implement decisions that have already been made.

They invest early in shared platform infrastructure: data lakes and warehouses, feature stores, catalogues, quality monitoring, governance and observability. Not one-off pipelines per project, but a proper refinery that serves the whole organisation.

And they build on foundations that can handle the scale and complexity of real enterprise data estates — structured tables alongside documents, images, logs and sensor data; on-premises alongside cloud and edge; batch pipelines alongside real-time streams.

That’s exactly what the Dell AI Data Platform is designed to support: a unified, modular foundation for storing, processing, governing and serving the data that modern AI workloads depend on — so data engineers can focus on building the refinery, rather than firefighting the infrastructure it sits on.


The Refinery Has to Work

The shift from rules-based systems to data-driven AI didn’t just give us more powerful software. It changed where the intelligence lives — and with it, what we need to invest in to make that software trustworthy.

When code was the truth, the bottleneck was engineers writing rules. When data is the truth, the bottleneck is the infrastructure that produces, refines, governs and delivers that data.

The refinery has to work. The pipelines have to be reliable. The fuel has to be clean. Everything downstream — every model, every decision, every output — depends on it.

And if you want to understand what happens when the refinery fails, that’s a story worth reading too.

The Magpie Effect: AI Possibilities vs Practical AI Solutions

An infographic titled 'The Magpie Effect' illustrating a magpie flying alongside various concepts related to AI strategies, constraints, and distractions. Includes labeled arrows pointing to elements like 'Distraction Vector', 'Shiny Object Response', and 'Golden Process - Critical Path'. Features a flowchart at the bottom indicating practical AI strategy.

Yesterday, I was sitting in a room listening to colleagues talk about the latest AI developments. New models. New capabilities. New promises about what AI will be able to do. It was energetic, enthusiastic, and genuinely well-informed.

And somewhere in the middle of it, a quiet thought surfaced: does any of this actually matter right here and right now?

It took me back to a book I read years ago when studying Business and Finance. The Goal by Eliyahu Goldratt. If you haven’t read it, it’s a business novel — a plant manager called Alex Rogo, a factory on the verge of closure, and a series of deceptively simple questions from an old professor that eventually save the business. The central insight is the Theory of Constraints: every system has a bottleneck. Find it, fix it, and the whole system improves. Ignore it, and no amount of optimisation elsewhere will save you.

Sitting in that room, I realised the AI industry has a Goldratt problem. It is endlessly fascinated by what is possible at the frontier, and not nearly interested enough in the bottlenecks that are quietly strangling real businesses today.

That is when the idea of the Magpie Effect crystallised for me.

There is a bird famously distracted by shiny things. It spots something glittering, drops whatever it was doing, and goes to investigate. Every few weeks, a new AI headline lands. A new model. A new capability. A new promise that this is the breakthrough that changes everything. Autonomous agents. Artificial General Intelligence. AI that writes code, runs your supply chain, manages your workforce. The glittering object rotates. The industry pivots. And somewhere in a conference room, a leadership team starts asking whether they should be doing that instead.

The Magpie Effect is quietly one of the biggest obstacles to real AI progress in business today. And Goldratt, I think, would have had little patience for it.

Goldratt was writing about factory floors. Machines, production lines, throughput, inventory. But swap the factory for a financial services firm, a retailer, or a healthcare provider, and the principle holds perfectly. The production lines just look different now. They are the workflows, the approval chains, the data pipelines, the customer journeys that run your business every single day. Constraints in those processes cost just as much as a jammed machine on a shop floor. They are just harder to see.


The Cost of the Chase

Chasing possibilities isn’t free. It has a price, and businesses are paying it in three currencies.

Time. Every pivot towards the next shiny object means restarting conversations, rewriting roadmaps, and pulling teams off work that was already delivering. The opportunity cost is real even when it’s invisible.

Money. Proof of concepts that never graduate. Vendor relationships built on promises. Platforms bought for futures that haven’t arrived. Science projects with no measurable return dressed up as innovation investment.

Momentum. Perhaps most damaging of all. Teams that are perpetually chasing the new never build the deep competency that comes from doing one thing properly, learning from it, and scaling it. They become generalists in hype rather than specialists in value.

The AI industry is not entirely to blame. Vendors need differentiation. Analysts need narratives. Media needs clicks. But the businesses that get swept up in it are making a choice — and there is a different choice available.


Practical AI Is Already Here. It’s Just Less Exciting.

Here is the uncomfortable truth: the AI that will genuinely move the needle for most organisations is not frontier. It is not exotic. And it is definitely not on the cover of a technology magazine.

It is the model that reduces manual data entry by 70%. The system that flags anomalies in financial transactions before a human would spot them. The tool that summarises a week of customer feedback into ten actionable themes in minutes. The scheduler that optimises field service routes and saves 15% on fuel costs.

None of those are science fiction. All of them are in production somewhere today. All of them have a measurable ROI. And none of them required waiting for the next frontier model to drop.

Practical AI solves a real business problem with available technology, accessible data, and a return you can put on a spreadsheet. That is not a consolation prize. That is the goal.


Start With Your Golden Process

Before you evaluate a single AI use case, ask one question: what is the one business process that, if it stopped tomorrow, the business would stop with it?

Not the most complex process. Not the most talked-about. The one that everything else depends on. The process that runs quietly in the background, holding the whole operation together. That is your Golden Process.

It might be order management. It might be claims processing. It might be production scheduling, customer onboarding, or logistics coordination. Every business has one. Most businesses have never explicitly named it.

Name it. Then look at it seriously.

Where does it slow down? Where does it depend on human heroics to keep running? Where does data get re-entered, reformatted, or chased across systems? Where do errors creep in? Those friction points are not just operational annoyances — they are your AI use case shortlist.

This is the starting point of a genuinely business-aligned AI strategy. Not a vendor briefing. Not a technology roadmap. Not a list of capabilities from the latest model release. The Golden Process and the friction within it.

The achievable business outcome — faster, cheaper, more reliable execution of the thing the business depends on most — is worth more than any amount of AI possibility. And it is almost always more achievable than it looks, because the process is already understood, the data already exists, and the ROI is not hypothetical.

Start there. Everything else follows.


How to Tell the Difference: The Practical AI Test

When an AI use case lands on your desk — from a vendor, from a consultant, from an enthusiastic team member who just watched a keynote — run it through these five questions.

1. What specific business problem does this solve? If the answer starts with “it could potentially…” or “imagine if we…” that is a possibility, not a solution. A practical use case has a named problem with a named owner.

2. Do we have the data to support it? AI without good data is not AI. It is noise with a marketing budget. Before you evaluate any use case, ask whether the relevant data exists, whether it is clean, and whether it is accessible. If the answer to any of those is no, the use case is not ready — regardless of how impressive the demo looked.

3. Do we have the skilled resources to support it? Good intentions and good data are not enough. Someone has to build it, maintain it, and own it when something goes wrong. That might be internal talent, a partner, or a combination — but the answer needs to be honest. An AI use case with no clear resource plan is not a use case. It is a wish list item.

4. Can we measure success? Define the KPI before you build anything. Handling time. Error rate. Cost per transaction. Customer satisfaction score. If you cannot articulate what winning looks like in a number, you are not running a business initiative. You are running an experiment with an open-ended budget.

5. Can this be in production within 90 days? Not perfect. Not scaled. But in the hands of real users, generating real outputs, against real data. If the answer is no, the scope is wrong or the foundations are not ready. Either problem needs solving before you proceed.

6. Would this still matter if nobody wrote about it? Strip away the hype. Imagine the use case is completely unglamorous — no AI branding, no press release, just a quiet improvement to a business process. Does it still make the list? If yes, it is probably worth doing. If the answer depends on how it sounds in a strategy presentation, be careful.


Make AI Boring. That Is the Win.

The organisations that are getting the most from AI right now are not the ones chasing the frontier. They are the ones that picked three or four use cases, proved the value, scaled what worked, and moved on to the next problem on the list. Quietly. Methodically. Without waiting for permission from the hype cycle.

They have made AI boring. And boring, in this context, is the highest compliment.

Boring means repeatable. Boring means trusted. Boring means it runs on a Tuesday afternoon without anyone noticing, because it has just become part of how the business works.

The magpies are still circling. The glittering objects will keep appearing. The organisations building durable capability are the ones that have learned to look away — and get back to the work that actually pays.


One Final Thought on Foundations

None of this works without getting data right. Practical AI is only as good as the data that feeds it. The organisations that move fastest are not necessarily the ones with the most advanced models — they are the ones whose data is clean, governed, and ready to use.

That is a less exciting conversation than whatever was announced at the last major AI conference. But it is the conversation that separates progress from theatre.

Data first. Practical use cases second. Everything else can wait.

At the end of The Goal, Alex Rogo doesn’t walk away with a new machine or a bigger budget. He walks away with a better question. Not “how do I optimise everything?” but “what is the goal — and what is stopping us from reaching it?” That shift in thinking is what saved the factory. It is the same shift that separates businesses building real AI capability from those still circling the next shiny object. Find your Golden Process. Name the constraints within it. Ask the right questions about where AI can remove them. The rest follows.


If you found this useful, the Getting AI Right First Time post covers the five steps to moving from AI experiments to durable enterprise capability.

AI Didn’t Come From Nowhere

Timeline infographic illustrating the historical development of artificial intelligence from the 1950s to today, including key eras such as symbolic AI, emergent algorithms, deep learning, and the transformer architecture.

I was running a workshop recently when something interesting happened. Mid-session, someone asked — genuinely, not rhetorically — when AI had actually started. The room went quiet. A few guesses landed around ChatGPT. One brave soul suggested maybe the 2010s.

Nobody said the 1950s.

And honestly? You can’t blame them. When the news cycle moves this fast, when the marketing is this loud, and when the rate of change feels genuinely relentless, it’s easy to assume we’re living through the birth of something. But AI doesn’t have a birth announcement. It has a long, slow build — one that stretches back decades, with foundations sunk deep in statistics and probabilistic reasoning long before the first neural network was trained or the first large language model wrote a sentence.

That context matters. Not as a history lesson for its own sake, but because understanding where AI came from changes how you think about where it’s going — and how you make decisions about it in your business today.

There’s a habit in technology — and AI coverage is particularly guilty of it — of presenting every new development as if it arrived fully formed from the future. A breakthrough. A paradigm shift. Something unprecedented.

AI is not that. And understanding why matters more than most people realise.


Where It Actually Started

The intellectual roots of AI don’t begin with computers. They begin with mathematics — specifically with statistics and probability theory developed long before anyone had written a line of code. Researchers working in the nineteenth and early twentieth centuries were already asking the question that defines AI today: how do we draw reliable conclusions from imperfect data?

Early artificial intelligence, when it finally emerged as a formal discipline in the mid-twentieth century, took a different path. The dominant approach was symbolic reasoning — hand-crafting rules, logic trees, and explicit instructions that told a computer exactly what to do in any given situation. If this, then that. Elegant in theory. Brittle in practice.

The real world, it turned out, doesn’t run on clean rules.


The Statistical Turn

During the latter half of the twentieth century, researchers began integrating statistical methods into computational models. Rather than encoding knowledge explicitly, they started asking whether machines could learn it from examples. It was a fundamental shift in approach — and it didn’t happen overnight.

The AI toolkit expanded steadily. Machine learning emerged as the discipline of teaching computers to find patterns rather than follow rules. Neural networks offered a loose computational approximation of how biological brains process information. Support vector machines provided a principled way to find decision boundaries in high-dimensional data. Decision trees made models interpretable. Ensemble methods — combining multiple models rather than relying on any single one — improved robustness. Each of these wasn’t a replacement for what came before. It was an addition. Another instrument in the orchestra.

Progress, though, remained constrained by two things: the computers weren’t fast enough, and the data wasn’t big enough.


When the Conditions Finally Met the Ideas

Both of those constraints began to ease significantly in the early twenty-first century. Processing power scaled. Storage costs collapsed. The internet generated datasets of a size that earlier researchers could only have imagined.

And something important happened: ideas that had existed in theoretical form for decades suddenly became practical.

Deep learning is the most visible example. The underlying concept — training neural networks with many layers to recognise complex patterns — wasn’t new. The mathematics had been worked out. The challenge had always been computational. When the hardware finally caught up with the theory, progress accelerated dramatically. Image recognition, speech processing, natural language understanding — fields that had plateaued for years began moving quickly again.

It would be easy to look at that moment and call it a revolution. It was, in a practical sense, transformative. But the ideas weren’t new. The infrastructure had finally arrived to make them real.


Iterative Refinement, Not Rupture

This is the part of the story that gets lost in most AI coverage: the field evolved through iteration, not through the wholesale replacement of one way of thinking with another.

Statistical theory didn’t displace symbolic reasoning entirely — it complemented and extended it. Deep learning didn’t make classical artificial intelligence obsolete — it sits alongside it. Modern practice draws from all of it: statistical theory, optimisation techniques developed over decades, and large-scale data processing infrastructure. The practitioner today reaches for whatever tool fits the problem.

Think of it like civil engineering. The Romans built arches. We still build arches. We also use steel, reinforced concrete, and computational modelling of load distribution. The new methods didn’t erase the old ones. They accumulated.


Why This History Matters

Understanding how artificial intelligence developed isn’t academic housekeeping. It directly shapes how you interpret what’s in front of you today.

The systems making headlines — the large language models, the image generators, the recommendation engines — are not wholly novel forms of intelligence. They are the product of long-standing theoretical ideas, finally meeting the computational infrastructure capable of realising them at scale. The mathematics has deep roots. The concepts have been refined across generations of researchers. The hardware and data are new. The thinking, largely, is not.

That’s not a diminishment. It’s a more honest and more useful frame.

When the next capability appears — and it will — the question worth asking isn’t where did this come from? The question is: what existing idea just found its moment?

Almost always, it had been waiting longer than the headlines suggest.

The Token Cost — New Line on the Spreadsheet


A budget breakdown chart outlining costs related to AI infrastructure, including license costs, infrastructure, staff costs, cloud compute, storage per GB, and a new line item for token costs, with arrows indicating actions required.

The Spreadsheet Never Lies

Back when I was an IT Manager, budget time was the one part of the job I genuinely dreaded.

I was technically biased — give me an infrastructure problem over a finance meeting any day. But the spreadsheet had to be built. So out it came, year after year. Rows and columns of licence costs, support contracts, hardware refresh cycles, staff costs, cloud compute, storage per GB. Every line item accounted for, justified, and defended.

Over the years that spreadsheet grew new rows. Cloud costs arrived and changed everything — suddenly you weren’t buying hardware, you were buying consumption. Then came storage costs per GB, virtual machine sprawl, networking costs, SaaS licensing, and the ongoing headache of software nobody was using but everyone was paying for.

Good IT management has always meant knowing what things cost. Not approximately. Precisely.

Now there is a new line to add to that spreadsheet.

Token cost.

But here is the thing. If you stop at the token line, you are optimising for the meter, not the mission.


Tokens 101 — What They Actually Are

Before the cost makes sense, the concept needs to.

A token is the basic unit an LLM uses to process text. Not a word. Not a character. Something in between — a chunk of text that the model reads, processes, and responds to.

When you type a message to a chatbot, the model doesn’t read it the way you wrote it. It breaks it into tokens first — fragments of words, whole words, punctuation, spaces — and processes each one in sequence. The response it generates is also built token by token, each one predicted from everything that came before.

A rough rule of thumb: one token is approximately four characters, or about three quarters of a word. A typical sentence of fifteen words is roughly twenty tokens. A detailed prompt of five hundred words is somewhere around six hundred and fifty tokens.

It adds up quickly. And every token processed — whether going in or coming out — carries a price.


Tokens Are a Meter, Not a Currency

There is a phrase doing the rounds right now. Tokens are the new currency of AI.

It is a neat soundbite. It is also wrong in all the ways that matter if you are trying to build serious AI capability.

Saying tokens are a currency is like saying you paid your electricity bill in kilowatt hours. You didn’t. You consumed kilowatt hours. You paid in money. The kilowatt hour is a unit of consumption — a meter reading, not a medium of exchange.

Tokens are exactly the same. They measure how much work a model is doing. They are the unit on which vendors calculate your bill. But they are not currency. They are consumption — and like every unit of consumption in IT, they carry a cost that needs understanding, governing, and optimising.

The organisations that treat tokens as a vanity metric — “we consumed X billion tokens last quarter!” — are optimising for the wrong number entirely.


The AI Factory and the Cost Behind Every Token

Dell and NVIDIA use the term AI Factory deliberately — because building AI capability at scale really does look like industrial infrastructure. Data pipelines, compute clusters, model serving layers, orchestration, guardrails. A factory for producing AI output at volume.

And like any factory, every unit of output carries a cost of production.

In an AI Factory, the token is the unit of output. And behind every token sits a cost stack most organisations never fully account for.

Infrastructure — GPU and accelerator time, CPU, RAM, networking, storage, cooling, power. Whether you see this directly or it is baked into a vendor’s price per thousand tokens, it is always there.

Model and platform — licensing for proprietary models, platform margin, optional add-ons for latency, SLAs, and private endpoints. Every provider has a margin sitting in the background of every token.

Data and training — models don’t appear from nowhere. Data acquisition, cleaning, fine-tuning, retrieval pipelines, continuous evaluation. All of it is part of the cost of making your tokens useful in your specific context, not just smart in general.

People — ML engineers, platform teams, application developers, security, compliance, prompt engineers. Labour is amortised over output. From a factory lens, every token carries a share of your people cost.

Guardrails and control — orchestration, content filters, safety checks, observability, caching, A/B testing. These are the conveyor belts and safety systems of your AI Factory. They rarely appear on a per-token price card. They always appear on your balance sheet.

The vendor gives you a clean price per thousand tokens. Your real cost per thousand tokens is considerably messier — and considerably higher.


From Token Cost to Outcome Cost

Here is where the conversation needs to move.

A token is a unit of cost. It is not a unit of value. And on its own, cost per token tells you almost nothing about whether your AI investment is working.

The number that actually matters is cost per outcome.

Swap abstract token consumption for something real: tokens per resolved support ticket. Tokens per sales proposal generated. Tokens per code review completed. Tokens per knowledge worker hour saved. Now you can build a unit economic view that means something.

Cost per outcome = (Tokens per outcome × fully loaded cost per thousand tokens) + overheads

Unit margin = Value per outcome − Cost per outcome

Once you see it this way, the conversations become sharper. A cheaper model per thousand tokens that requires three times the tokens per outcome is not a saving. A use case that looks expensive in tokens but delivers enormous value per outcome is not a problem. A system regenerating the same content repeatedly because nobody implemented caching is a straightforward fix hiding in plain sight.


The Levers: Token Productivity in the AI Factory

If tokens are the output of your AI Factory, token productivity is your primary optimisation lever.

Use the right model for the job. Not everything needs your largest, most capable model. Smaller, cheaper models handle classification, routing, and simple transforms well. Reserve the heavy models for genuinely complex reasoning. A tiered approach — cheap model first, escalate only when needed — can dramatically change your cost per outcome without touching quality.

Optimise prompts and context. Long system prompts and bloated context windows feel powerful. They are also expensive. Strip repetition, keep only relevant context, use structured inputs where possible. Every unnecessary sentence in a prompt is scrap material on the factory floor — and in a high-volume system, scrap accumulates fast.

Cache intelligently. A significant proportion of enterprise AI workloads are repetitive — similar questions, standard documents, known sub-tasks. Response caching, retrieval caching, and partial caching of intermediate steps reduce tokens per outcome without any loss of quality. It is one of the highest-return optimisations available and one of the most consistently overlooked.

Design around outcomes, not demos. Demos optimise for the impressive moment. Factories optimise for throughput and margin. Start from the business outcome, the current human cost of achieving it, and the target cost with AI. Then design the system backwards from that constraint — not forwards from whatever the latest model happens to be capable of.


Token Cost as a Governance Question

This is familiar territory for anyone who has managed cloud costs or software licensing.

Token consumption is a shared resource. Different business units, different applications, and different use cases will consume it at different rates and generate very different outcomes per token. Without visibility into that consumption — tracked by application, by business unit, by use case — you have no basis for budgeting, no mechanism for chargeback, and no way to identify where usage is growing faster than the value it is generating.

A note on agentic AI: if your organisation is moving into agentic deployments — systems that reason across multiple steps, use tools, retrieve information, and check their own work — the token cost profile changes significantly. A standard chatbot interaction might consume a few hundred tokens. An agentic workflow handling the same underlying task can consume tens of thousands. Model it separately. Budget it separately. The capability gain can be substantial, but the consumption profile is a different order of magnitude.


Optimising for the Mission

Back at that budget spreadsheet, the discipline was always the same. Know what you consume. Know what it costs. Know who is consuming it. And know what value it is generating.

Tokens deserve exactly that discipline. Not because they are a currency. Because they are a cost — the most visible signal of the underlying economics of your AI Factory.

The token line on the bill matters. But the executives asking “what is our token budget this year?” are asking the wrong question.

The right questions are these: Which AI-enabled outcomes matter for our business? What is our target cost per outcome? What mix of models, infrastructure, and data do we need to get there? And how do we measure value per outcome — not just tokens consumed?

Tokens are how you keep score in the background. Outcomes are why you are playing.

If your AI strategy stops at tokens, you are optimising for the meter, not the mission.

AI is Closer Than You Think: Machine Learning Part 4

Flowchart illustrating the agent feedback loop in reinforcement learning, featuring steps: Observe State, Choose Action, Receive Reward, and an environment section with 'Try -> Fail -> Adjust' notation.

Game Over

Miss Smith taught me to predict the future with a straight line.

A shoebox of CDs taught me to find patterns without labels.

Reinforcement learning came from somewhere else entirely.

It came from losing.

The game was Doom. Dark corridors, demons around every corner, no manual, no walkthrough, no one to copy. Just a Marine on a screen, a keyboard under my fingers, and the immediate brutal feedback of getting it wrong.

I died constantly. But somewhere in that cycle of failure, something started to form. Move left here. Don’t open that door without backing up first. The shotgun beats the pistol in a corridor. The game never explained any of it. It just reacted — survival or slaughter — and I adjusted.

Try something. Die. Try something slightly different. Die slightly later. Somewhere in that loop, a strategy emerged.

That feedback loop, as it turns out, is one of the most powerful ideas in machine learning.


The Third Way of Learning

In Part 2, supervised learning gave the machine a cheat sheet — labelled examples with known answers to learn from. In Part 3, unsupervised learning pulled the cheat sheet away and let structure emerge from the data itself.

Reinforcement learning is different from both. There are no labelled examples to study and no static dataset to mine. Instead, there is an agent, an environment, and a simple but profound arrangement: act, observe the consequences, and gradually learn a strategy that leads to better outcomes over time.

Not “get this prediction right.” Not “find what naturally groups together.” But — do well in the long run.

That shift from a single correct answer to long-term cumulative reward is what makes reinforcement learning feel unlike anything else in the machine learning story.


The Loop

The mechanics are simple enough to hold in your head.

An agent observes the current state of its environment. It chooses an action. The environment responds — a reward, a penalty, or silence. The agent updates its understanding of which actions tend to lead where, and chooses again.

Run that loop millions of times and something remarkable happens. What started as near-random behaviour gradually sharpens into strategy. Not because anyone defined the right moves in advance, but because the feedback itself did the teaching.

This is how DeepMind’s AlphaGo learned to defeat the world’s best human players using strategies no human had ever conceived. Not by studying a rulebook, but by playing millions of games against itself, adjusting after every one. The data didn’t come from a spreadsheet. It came from experience.


Rewards, Not Labels

In supervised learning the feedback is immediate and precise. You predicted £350,000. The true answer was £320,000. Here is exactly how wrong you were.

In reinforcement learning it rarely works like that.

Think of teaching a dog a new trick. You don’t label each tiny movement as right or wrong. You reward certain sequences of behaviour — sit, stay, come — and ignore or discourage others. Over time the dog figures out which actions tend to lead to treats, even though nobody narrated the journey step by step.

Reinforcement learning agents learn in the same way. They explore — often taking random actions just to see what happens. When they stumble onto something that works, the algorithm quietly reinforces the decisions that led there. When they hit bad outcomes, it weakens them.

The hard part is that rewards can be a long time coming. In a game, the winning move might trace back to a decision made hundreds of steps earlier. In a supply chain, a choice that looks costly today might pay off weeks later. Working out which actions deserve the credit — or the blame — across a long sequence of decisions is one of reinforcement learning’s central challenges. And one of its most important unsolved problems.


Closer to Everyday Life Than You Think

The famous examples tend to involve games and robots — AlphaGo, robotic arms, drones balancing in turbulence. These make good headlines. But reinforcement learning is also quietly at work in less dramatic places.

The route your sat-nav recalculates in real time. The dynamic pricing that adjusts what you’re shown based on demand and your behaviour. The recommendation engine that doesn’t just respond to what you clicked last, but learns how to keep you engaged across an entire session. In each case, a sequence of decisions is being optimised for a long-term outcome — not a single prediction, but a strategy playing out over time.

At enterprise scale, the same principles are starting to touch logistics, energy management, and operational scheduling — systems where the cost of a bad sequence of decisions is measured not in lost points but in real money and real consequences. (The data infrastructure that makes that possible is something we explore in our [Data and AI series].)


When the Game Goes Wrong

When I was losing at video games, the stakes were low. The worst outcome was a game over screen and a bruised ego. I could experiment freely because nothing real was at risk.

Reinforcement learning systems don’t always have that luxury — and when the reward is defined badly, the results can be deeply strange.

Agents are remarkably good at finding ways to maximise whatever score they’ve been given, including ways nobody anticipated and nobody wanted. A robot that learns to exploit a bug in the simulation. An ad system that maximises clicks by surfacing content that is technically engaging but clearly not what anyone intended. The agent isn’t being clever or malicious. It’s doing exactly what it was asked. The problem is that what it was asked and what was actually wanted weren’t quite the same thing.

This is why reinforcement learning often starts in simulation, with careful constraints before anything touches the real world. And it’s why the design of the reward itself is not a technical afterthought — it’s one of the most consequential decisions in the whole system.


One Is About Answers. One Is About Patterns. One Is About Behaviour.

Supervised learning, unsupervised learning, and reinforcement learning aren’t separate islands. In practice they layer and interweave — reinforcement learning agents often use supervised models inside themselves, predicting future rewards or modelling parts of their environment. Unsupervised techniques help them compress complex states into something manageable.

But as a map of the territory, the distinction holds.

Supervised learning is about answers. Unsupervised learning is about patterns. Reinforcement learning is about behaviour — learning how to act in an environment to maximise what matters over time.


Trial, Error, and Responsibility

When I finally stopped dying in that game, it wasn’t because someone handed me the solution. It was because the loop — try, fail, adjust, try again — had quietly built something that worked.

Reinforcement learning systems learn the same way. Which means they will find strategies we didn’t anticipate, shortcuts we didn’t design, and solutions we didn’t know were possible. Sometimes that’s extraordinary. Sometimes it’s a warning.

The trial-and-error loop doesn’t go away. But once you give an agent the power to act in the world, the design of its rewards, constraints, and environment becomes an ethical question as much as a technical one.

Because players, as I learned the hard way, will do whatever it takes to win the game you set in front of them — whether or not it’s the game you thought you were designing.


In the next post, we step back from the mechanics and look at the bigger picture — where these three ways of learning meet the real world, and what it means to build systems that don’t just work, but work well.

AI is Closer Than You Think: Machine Learning Part 3


A diagram illustrating unsupervised learning with three distinct data segments labeled A, B, and C, showcasing clusters identified by a K-means algorithm, with annotations about input and output data patterns.

The Shoebox

Before streaming, my music lived on CDs. No playlists, no algorithms, no recommendations. Just a pile of discs in a shoebox and an afternoon to fill.

Every so often I’d try to organise it. Not alphabetically — that always felt a bit lifeless — but by feel. The driving albums. The get romantic albums. The dance in the kitchen albums when no ones home.

Nobody told me what those categories were. There were no labels to copy, no genres printed on the spine that captured what I actually meant. I just listened, paid attention, and started to notice what belonged together.

Without knowing it, I was doing the analogue version of something machines now do at industrial scale.

It’s called unsupervised learning. And it might be the most quietly remarkable thing in the whole machine learning story.


Pull the Labels Off

In Part 2, we explored supervised learning — where a model studies thousands of labelled examples, finds the pattern connecting inputs to outputs, and uses it to predict something new. The label is the teacher. The cheat sheet is always there.

Unsupervised learning pulls the cheat sheet away entirely.

The data arrives with no correct answers attached. No column marked “right” or “wrong.” No target to aim for. Just raw information and the instruction: see what you can find.

This is where things get genuinely interesting. Because some of the most valuable patterns in the world haven’t been labelled yet — not because nobody cared, but because nobody knew they were there.


Two Ways of Seeing in the Dark

Unsupervised learning tends to show up in two broad forms.

The first is clustering — the shoebox problem. You’re not told what the groups should be. You look at what’s similar and let the structure emerge. Machines do this with customers, documents, transactions, sensor readings — treating each example as a point in space and finding the natural islands of similarity that form within it. The algorithm finds the groups. You decide what they mean.

The second is dimensionality reduction — a way of finding what’s really going on underneath data that looks impossibly complex. Real-world datasets can have hundreds of variables. Dimensionality reduction compresses them down into a smaller number of hidden patterns — the underlying forces that explain most of what’s happening. In music data those forces might feel like energy, mood, or tempo. In customer data they might feel like loyalty, price sensitivity, or seasonality. Nobody defines those dimensions in advance. The data reveals them.

In both cases, the machine finds the structure. We supply the language.


From Shoebox to Segments

Step out of the shoebox and into something closer to the real world: a retailer with fifty thousand customers and no idea how to talk to them differently.

The data is all there — purchase frequency, spend, product categories, response to promotions. What isn’t there is a neat label saying budget buyer or loyal advocate or one-off bargain hunter. Those categories, if they exist at all, live only in someone’s intuition.

A clustering algorithm doesn’t need the labels. It looks at how customers behave, measures the distances between them in that behaviour space, and finds the natural groupings that form. What comes out the other side isn’t a category from a textbook — it’s something that emerged from the data itself.

These customers buy rarely but spend significantly when they do. These buy constantly but only when there’s a promotion. These arrived in a burst eighteen months ago and then quietly disappeared.

Nobody wrote those segments. The data found them. And once you can see them, you can act on them — which is exactly the kind of insight that scales from a single analyst’s instinct to an enterprise decision engine. (The infrastructure that makes that scale possible is something we explore in our [Data and AI series].)


Learning Without a Score

With supervised learning, the feedback loop is clean. Predict. Compare to the label. Adjust. The model always knows how wrong it was.

In unsupervised learning there is no label, so there is no score in that sense. The algorithm isn’t trying to match a known answer — it’s trying to find the most meaningful structure it can, given only the data in front of it.

That shifts what “doing well” means. A good clustering result isn’t one that matches a predefined answer. It’s one where the groups are genuinely distinct, internally coherent, and — crucially — useful to the people working with them. An elegant algorithm that produces segments nobody can act on has failed, regardless of its technical score.

This is why unsupervised learning asks more of the humans involved, not less. The machine surfaces the pattern. The judgement about whether that pattern matters still belongs to us.


When It Goes Wrong

Unsupervised learning can go wrong in ways that are subtler than supervised learning — and sometimes harder to catch.

Given enough freedom, almost any dataset can be divided into clusters. That doesn’t mean those clusters are real. The most dangerous outcome isn’t a model that fails visibly — it’s one that produces beautiful, confident-looking segments that collapse the moment someone who knows the business actually looks at them.

There’s also the temptation to storytell on top of weak patterns. Once you have groups, it’s very easy to stand back and say: ah yes, these are clearly our innovators, these are our laggards. Sometimes that’s true. Sometimes it’s one variable you forgot to account for, dressed up in a compelling narrative.

And the ethical dimension doesn’t disappear just because the labels do. If certain groups are underrepresented in your data, the patterns that emerge will reflect that absence — quietly, confidently, and without a warning sign. As we explored in Part 1 and Part 2, the quality of data is never just a technical question.


Siblings, Not Rivals

It’s tempting to think of supervised and unsupervised learning as separate worlds. One with labels, one without. One for prediction, one for exploration.

In practice they constantly work together. Unsupervised methods often do the groundwork — simplifying, exploring, revealing structure — that makes a supervised model possible. Supervised labels can be tested against unsupervised findings to check whether what you think the categories are matches what the data actually shows.

Miss Smith’s straight line and the shoebox of CDs aren’t different stories. They’re different chapters of the same one.


Closer Than You Think

When I was sorting that shoebox, I wasn’t thinking about algorithms. I was just following my ears — noticing what felt similar, grouping it, deciding what the groups meant.

Today, unsupervised learning does that for millions of customers, billions of web events, and oceans of sensor data that no human could sort by feel. No labels. No cheat sheet. Just structure, pulled out of the noise.

In the next post, we’ll look at reinforcement learning using a computer game analogy.

Because that’s when it stops being interesting and starts being important.

AI is Closer Than You Think: Machine Learning Part 2

Graph illustrating supervised learning in machine learning, showing a relationship between feature input and price output, with labeled axes for predictions and adjustments, and annotations for training data and patterns found.

Miss Smith’s Straight Line

Miss Smith did a number on me. Back at school the one class I liked was math. I liked the logic and solving puzzles through patterns. My teacher, Miss Smith was awesome. One afternoon she drew a straight line through a cloud of scattered points on graph paper. And something clicked.

“Wait. I can predict the future with this?”

That moment — without knowing it — was my introduction to supervised learning. The idea that you can study past examples, find the pattern underneath, and use it to make confident predictions about something you’ve never seen before.

It’s the same principle that prices houses, filters your inbox, approves your mortgage, and — as I discovered on a campsite in Wales — catches fraud on your bank card in real time.

In Part 2 of my Machine Learning series, I explore how supervised learning actually works. Not the maths. Not the code. The idea — and why it matters.

Because Miss Smith’s straight line didn’t stay on graph paper. It scaled up into some of the most consequential technology running in the world today.


The Cheat Sheet, Revisited

In Part 1, I introduced supervised learning as the approach where machines learn from labelled data — a cheat sheet of examples where the correct answer is already known. Now let’s get under the bonnet.

The best way to understand supervised learning is through a problem everyone has an instinct for: house prices.

Imagine you’re trying to predict what a house will sell for. You have years of sales data — hundreds of houses, each one described by its size, number of bedrooms, age, location, and a dozen other details. And crucially, you know what each one actually sold for.

That dataset is your cheat sheet. The model’s job is to study it — not to memorise it, but to find the pattern underneath. The relationship between what a house is and what it’s worth. Once it has that pattern, you hand it a house it has never seen before, and it makes a prediction.

That’s supervised learning. Past labelled examples, used to make confident predictions about the future.

Miss Smith would recognise it immediately.


Two Ways the Problem Can Look

Supervised learning shows up in two broad flavours, depending on what you’re trying to predict.

The first is regression — predicting a number. House prices are a regression problem. So is forecasting next week’s energy demand, estimating how long before a machine needs maintenance, or predicting a patient’s recovery time. The output is a value on a continuous scale.

The second is classification — predicting a category. Is this email spam or not? Is this transaction fraudulent or legitimate? Is this scan showing signs of disease? The output is a label, a decision, a bucket.

The mechanics underneath both are remarkably similar. In each case, the model studies labelled examples, finds the pattern connecting inputs to outputs, and applies that pattern to new data it hasn’t seen before.

What changes is the shape of the answer.


Learning from Mistakes

Here’s what makes supervised learning work — and what makes it genuinely clever.

The model doesn’t start knowing anything. It begins with a guess — an initial, probably terrible attempt at the pattern. For a house price model, that first guess might be wildly wrong. A three-bedroom terrace in Manchester valued like a penthouse in Chelsea.

But the model knows it’s wrong, because it has the actual sale price right there in the training data. So it adjusts. It tweaks its understanding of which features matter and by how much. Then it guesses again. Checks again. Adjusts again.

Predict. Compare. Adjust. Repeat.

Run that loop across thousands of examples, and something remarkable happens. The model stops guessing and starts understanding. Not because anyone explained the relationship between a house and its value — but because the data did.

This is the same fundamental shift we explored in Part 1. Logic didn’t drive the output. Data did.


When It Goes Wrong

Supervised learning is powerful. It’s also fallible in ways worth understanding.

The most common trap is overfitting — when a model studies its training data so closely it starts memorising quirks rather than learning patterns. It performs brilliantly on examples it’s seen before and poorly on anything new. Like a student who memorises past exam papers word for word, then struggles the moment a question is phrased differently.

The fix is discipline. You always hold back a portion of your data — a test set the model never sees during training — and use it to check whether what the model learned actually generalises.

The deeper trap is bad data. Supervised learning is only as honest as the labels it learns from. If your training data reflects historical biases — in lending decisions, hiring patterns, medical diagnoses — the model will learn those biases and reproduce them at scale. Quietly. Confidently. At speed.

This is why the quality of data isn’t just a technical question. It’s an ethical one. In our [Data and AI series], we explored what it takes to transform raw data into something trustworthy. In supervised learning, that trustworthiness isn’t optional — it’s foundational.


The Straight Line, Scaled Up

Miss Smith’s straight line through a cloud of points is, technically, linear regression — one of the simplest supervised learning models there is. It assumes the relationship between inputs and output can be captured in a weighted sum. Each feature gets a number, and the model learns which numbers make the predictions most accurate.

It sounds simple. In some contexts it’s all you need. In others — where the relationships are non-linear, where the data is images or language or sound — you need something more powerful. Decision trees. Random forests. Gradient boosting. And at the frontier, the neural networks that underpin deep learning.

But here’s what matters: no matter how complex the model, the supervised learning loop doesn’t change.

Past labelled examples. A pattern found in the data. A prediction made about something new.

Miss Smith’s straight line. Just drawn with considerably more dimensions.


Closer Than You Think

Supervised learning isn’t an abstract concept living in a textbook. It priced the house you live in. It filtered the spam you never saw this morning. It flagged the fraudulent transaction on your card — which, as it happens, I know from personal experience it does rather well on a campsite in Wales.

In the next post, I’ll look at unsupervised learning — what happens when there’s no cheat sheet, no labelled answer, and the machine has to find the pattern entirely on its own.

The results, as it turns out, are often the most surprising of all.

AI – Closer than you Think: Machine Learning Part 1

A diagram illustrating machine learning systems with three branches: Supervised Learning connected to a smartphone, Unsupervised Learning linked to headphones, and Reinforcement Learning associated with a location marker on a blueprint-style background.

A Text Message in Wales

Last summer I was camping in Wales. Sunny coast, fresh air, about as far from a city as you can reasonably get on a British weekend. My phone buzzed. A text from my bank — an automated alert. My credit card may have been compromised.

I called the bank hotline. Sure enough, while I was sitting by the Welsh coast, my card was apparently having a very different kind of day. Someone had just spent £150 at a noodle bar in London.

£150. At a noodle bar. Who eats that much?

It was fraud. But here’s the part that stayed with me. My bank didn’t catch it because a human spotted something suspicious. They caught it because a system had been quietly watching — not just my transaction, but the pattern of it. My location. My usual spending habits. The gap between where I was and where my card suddenly appeared. The amount. The time. The type of merchant.

No rule said flag this transaction. The data said it.

That moment — sat in a field in Wales, grateful for an automated text message — is what this series is about. Machine learning isn’t a lab experiment or a boardroom buzzword. It was working for me on a campsite, in real time, without me ever thinking about it.

It’s closer than you think.

Machine Learning — It’s Not What You Think

We’ve been programming computers for decades. You tell it what to do, it does it. Simple. A calculator adds numbers because a programmer wrote the rule: take this input, apply this operation, return this output. The logic is explicit, predetermined, and entirely human-authored.

Machine learning breaks that contract.

Instead of writing rules, you feed the system data — lots of it — and let it figure out the patterns itself. The output isn’t driven by what a programmer anticipated. It’s driven by what the patterns the data reveals. The programmer’s job shifts from writing the answer to building the environment where the answer can be discovered.

Think of it this way. If you wanted a traditional program to identify a cat in a photo, you’d have to write rules: four legs, pointy ears, whiskers, tail. But what about a cat curled into a ball? Or half-hidden behind a sofa? The rules break down almost immediately. There are too many variables, too many edge cases. No programmer can anticipate them all.

Machine learning takes a different path. Show it ten thousand photos of cats — and ten thousand photos of things that aren’t cats — and it learns to recognise the difference itself. Not because anyone told it what a cat looks like, but because the data did.

This is the fundamental shift. In traditional programming, logic drives output. In machine learning, data drives output. And that single distinction changes everything about how we build software, what problems we can solve, and — as we’ll explore throughout this series — how quietly and completely it has already woven itself into everyday life.

(If you’ve been following our [Data series], you’ll recognise a familiar theme here: data isn’t exhaust. It’s the raw material. In machine learning, it’s more than that — it’s the teacher.)


Teaching Machines — With and Without a Cheat Sheet

Not all machine learning works the same way. At its core, there are two broad approaches — and understanding the difference helps explain why ML can tackle such a surprisingly wide range of problems.

Supervised Learning: Learning with a Cheat Sheet

In supervised learning, the machine is trained on data that has already been labelled. Someone — or something — has done the groundwork of tagging the answers before the learning begins.

Think of it like a student revising with a marked answer sheet. The model sees thousands of examples where the correct answer is already known, learns the patterns that connect inputs to outputs, and then applies that knowledge to new, unseen data.

Spam filters are a classic example. Years of emails labelled spam and not spam have trained models to recognise the telltale patterns — certain words, unusual senders, suspicious formatting — before a single message reaches your inbox. You never see the decision being made. You just find the junk mail already sorted.

The same principle powers fraud detection on your bank account, product recommendations on Amazon, and the autocorrect quietly fixing your typos as you type.

Unsupervised Learning: Finding Patterns Nobody Labelled

Unsupervised learning starts with no cheat sheet at all. The data arrives unlabelled, and the model’s job is to find structure within it — groupings, patterns, anomalies — without being told what to look for.

This is where things get interesting. Rather than confirming what we already know, unsupervised learning can surface things we didn’t expect.

When Spotify groups listeners into taste clusters — not by genre, but by nuanced listening behaviour — nobody sat down and defined those clusters in advance. The model found them in the data. The same happens when retailers segment their customers, not by demographics they collected, but by purchasing patterns the data itself revealed. The insight emerges from the data rather than from a human hypothesis.

Reinforcement Learning: Learning by Doing

There’s a third approach — and it works nothing like the other two.

Reinforcement learning doesn’t use labelled data, and it doesn’t mine an existing dataset for hidden patterns. Instead, it learns through trial and error, in real time, guided by reward and consequence. The model takes an action, observes the result, and adjusts its behaviour to maximise a reward signal over time.

It’s closer to how humans learn a skill than how we study for an exam. A child learning to ride a bike isn’t handed labelled examples of successful balance. They fall, adjust, try again, and gradually internalise what works. Reinforcement learning follows the same logic — just at machine speed and scale.

The most famous examples involve games. DeepMind’s AlphaGo didn’t study a rulebook and wasn’t shown the right moves. It played millions of games against itself, learning which strategies led to victory and which didn’t. The result was a system that defeated the world’s best human players using approaches no human had considered.

But reinforcement learning isn’t confined to games. It’s how autonomous vehicles learn to navigate complex traffic scenarios, how robotics systems develop precise physical movements, and — closer to home — how recommendation engines quietly refine themselves based on what you watch, skip, pause, or replay. Every interaction is a signal. Every signal shapes the next decision.

Why it Matters

Supervised learning is powerful when you know what you’re looking for. Unsupervised learning surfaces what you didn’t know to look for. Reinforcement learning figures out how to act in a world where the right answer only becomes clear over time.

Most sophisticated ML systems draw on all three. And as we’ll see in the sections ahead, these approaches combine in ways that have quietly reshaped the texture of everyday life.

(In our [Data and AI series], we explored how raw data becomes usable through transformation and engineering. Here, that refined data becomes the curriculum — the material from which machines learn.)


Closer Than You Think

Here’s the thing. You don’t need to understand any of this to already be living with it.

This morning, before you’d even had a coffee, machine learning was already at work. Your phone unlocked when it recognised your face — supervised learning, trained on thousands of facial images. Your music app queued something you hadn’t heard before but somehow knew you’d like — unsupervised learning, grouping your taste with people you’ll never meet. And the route your maps app chose to avoid that unexpected roadblock? Reinforcement learning, continuously updated by millions of live journeys happening in parallel.

No rules were written for any of those moments. No programmer anticipated your face, your mood, or that particular stretch of roadwork. The data did the work.

That’s machine learning in everyday life. Not a distant technology. Not a lab experiment. Something already woven into the small, unremarkable moments that make up your day — quietly learning, adapting, and improving, whether you notice it or not.

In the posts ahead, we’ll pull back the curtain on more of those moments — and start to explore what happens when the same principles scale from your pocket to the enterprise.

Having my Cake and Eating it

A freshly baked round vanilla cake cooling on a wire rack, displaying a golden-brown top with a slight crack.

Last year, during my machine learning studies, I was handed two datasets: one detailing cake ingredients, one recording baking outcomes. My task was to find the optimal ingredient ratios — the mathematical sweet spot that balanced flavour, consistency, cost and more simultaneously.

The optimisation worked. After running the model, I had my answer: five numbers, each between zero and one, representing the theoretically perfect proportions of flour, sugar, butter, eggs and milk.

The problem? Ratios are just numbers. You can’t bake numbers.

So I did something that felt slightly absurd at the time: I connected my ML model to generative AI — specifically to Claude — and asked it to take those optimised ratios and turn them into a recipe a human could actually follow. In the voice of Gordon Ramsay, if possible.

It worked. Rather well, actually.

This post is the write-up of that experiment. It’s the story of what I built, why it works, and what it taught me about the difference between two distinct but complementary types of AI: machine learning that optimises, and generative AI that communicates.

ML + GenAI = Cake.


What does the perfect cake look like?

Not aesthetically — mathematically. If you had to define “perfect” as a set of competing objectives — flavour, consistency, calorie density, ingredient cost, waste — and then search across every possible combination of flour, sugar, butter, eggs and milk to find the recipe that satisfies all of them simultaneously, how would you do it?

You could try every combination. But five ingredients, each varying continuously between zero and one, produces an effectively infinite search space. Brute force is out.

You could guess, bake, taste, adjust, and repeat. That’s essentially what bakers have done for centuries. It works, but it’s slow, expensive, and entirely dependent on the baker’s intuition about which direction to adjust next.

Or you could do what this project does: treat cake baking as a five-dimensional optimisation problem, use a machine learning technique called Bayesian Optimisation to navigate that space intelligently, and then hand the winning recipe ratios to Claude and ask Gordon Ramsay — or Mary Berry, or Jamie Oliver — to turn them into something you’d actually want to bake.

The cake is a vehicle. What we’re really exploring is one of the most elegant and underappreciated ideas in machine learning: how to search for the best answer when testing every answer is too expensive.


The Problem With Trying Everything

In earlier posts in this series, we described machine learning as finding patterns that connect inputs to outputs. Most of the techniques we’ve explored — supervised learning, attention mechanisms, data pipelines — are about learning from large amounts of existing data.

Bayesian Optimisation is different. It’s designed for situations where generating data is costly.

Imagine you’re developing a new drug formulation. Each trial costs £50,000 and takes six months. Or you’re tuning the hyperparameters of a large neural network — each training run takes days of GPU time. Or, more deliciously, you’re trying to find the objectively best cake recipe and you only have so much butter.

In all of these cases, you can’t afford to try everything. You need a strategy for choosing which experiment to run next — one that makes the most of everything you’ve already learned.

That’s exactly what Bayesian Optimisation does.


The Surrogate Model: Building a Mental Map

The key idea behind Bayesian Optimisation is the surrogate model — a simpler, faster model that approximates the expensive function you’re trying to optimise.

Think of it like this. You’re a food critic trying to find the best restaurant in a city you’ve never visited. You can’t eat at every restaurant — there are thousands of them and you only have a week. But after visiting ten or fifteen, you start to build a mental map: good restaurants tend to cluster in certain neighbourhoods, certain cuisine types, certain price ranges. You use that mental model to decide where to try next — not randomly, but informed by what you’ve already learned.

In our cake system, the surrogate model is a Gaussian Process — a mathematical framework that, given the recipes we’ve already evaluated, produces two things for any recipe we haven’t tried yet: a predicted score (our best guess at how good it would be) and a measure of uncertainty (how confident we are in that prediction).

The Gaussian Process is fitted to a starting dataset of twenty recipes — a mix of traditional ingredient ratios and random samples. From there, it builds a probabilistic map of the entire five-dimensional recipe space, identifying where good recipes are likely to live and, crucially, where we don’t yet know enough to say.


The Acquisition Function: Deciding What to Try Next

Having a map is useful. But which location on the map do you visit next?

This is the job of the acquisition function — specifically, Expected Improvement (EI). It answers a deceptively simple question: given everything we know so far, which untested recipe is most likely to beat our current best?

Expected Improvement balances two competing instincts:

Exploitation — test recipes in regions we already know are promising. If we’ve found that high butter and moderate sugar scores well, try nearby variations.

Exploration — test recipes in regions we’re uncertain about. There might be something surprising in a corner of the space we haven’t visited yet, and if we only exploit what we know, we’ll never find it.

The EI function quantifies this trade-off mathematically. Regions with high predicted scores get attention because they’re likely to be good. Regions with high uncertainty get attention because the potential upside is large. The next recipe to evaluate is the one that maximises expected improvement, balancing both factors simultaneously.

# Simplified Expected Improvement calculation
def expected_improvement(mean, std, best_so_far):
improvement = best_so_far - mean # how much better could this be?
Z = improvement / (std + 1e-9) # normalised by our uncertainty
ei = improvement * norm.cdf(Z) + std * norm.pdf(Z)
return ei

After evaluating the suggested recipe, the Gaussian Process updates its map, the acquisition function identifies the next best candidate, and the cycle repeats — for forty to fifty iterations in our system. Each iteration makes the map more accurate and the search more targeted.

This is Bayesian Optimisation: an intelligent, iterative loop that learns from every experiment to make the next one count more.


Five Dimensions, Five Objectives

Our recipe space has five ingredients — flour, sugar, butter, eggs, milk — each represented as a continuous ratio between zero and one. Every point in that five-dimensional space is a candidate recipe.

Every candidate recipe is evaluated against five competing objectives:

  • Flavour — rewarding an optimal sugar-to-butter ratio and sensible flour range
  • Consistency — rewarding an appropriate flour-to-liquid ratio
  • Calories — penalising deviation from a target calorie density
  • Waste — penalising extreme quantities at either end of the range
  • Cost — penalising deviation from a target recipe cost, using weighted ingredient prices

Each objective contributes a penalty score. The goal is to find the recipe whose total penalty is closest to zero — the theoretical point of perfect balance across all five dimensions simultaneously.

Real-world baking variability is simulated by adding a small amount of random noise to each evaluation. This is deliberate: in practice, optimisation problems are rarely perfectly smooth, and a robust system needs to navigate noise rather than overfit to it.

The best recipe found after fifty iterations is not a human-readable recipe. It’s a set of five numbers — ratios that represent an optimal point in an abstract mathematical space. Which is where the second half of the system comes in.


From Ratios to Recipes: Handing Off to Claude

Once optimisation is complete, the winning ratios are passed to the Claude API with a prompt that does three things: scales the abstract ratios into real measurements for a standard nine-inch cake, adds the supporting ingredients any real recipe needs (baking powder, salt, vanilla), and writes the whole thing in the voice of a specific chef persona.

The same optimised ratios, rendered six different ways:

Jamie Oliver gets enthusiastic and accessible — “roughly a good big mug of flour,” “a proper glug of vanilla.” Gordon Ramsay gets precise and demanding — grams, temperatures, technique. Mary Berry gets gentle and reassuring — “fold carefully,” “don’t rush the creaming stage.” The German version arrives in Bavarian style. The Mandarin version in Chinese.

The mathematical optimisation and the language generation are entirely separate concerns. The optimiser doesn’t know anything about chefs. Claude doesn’t know anything about Gaussian Processes. Each does what it does best, and the handoff between them is just five numbers and a prompt.

prompt = f"""
You are {chef_persona}. Convert these optimised ingredient ratios into a
complete recipe for a 9-inch cake. Ratios: flour={ratios[0]:.3f},
sugar={ratios[1]:.3f}, butter={ratios[2]:.3f}, eggs={ratios[3]:.3f},
milk={ratios[4]:.3f}. Scale to real measurements, add standard supporting
ingredients, and write in your characteristic voice.
"""

This pattern — optimise mathematically, communicate humanly — is increasingly how AI systems work in practice. The model doesn’t replace the domain expertise. It translates it.


What Bayesian Optimisation Is Really For

The cake is fun. But the technique matters far beyond baking.

Bayesian Optimisation is one of the most widely used approaches for hyperparameter tuning — the process of finding the best configuration for a machine learning model itself. Learning rate, batch size, network depth, regularisation strength: these are the knobs that determine whether a model trains well or wastes weeks of GPU time producing something mediocre. Testing every combination is prohibitively expensive. Bayesian Optimisation finds good configurations in a fraction of the trials that random or grid search would require.

It’s used in drug discovery, materials science, engineering design, and financial model calibration — anywhere the cost of a single experiment is high and the search space is large.

The core insight is always the same: you don’t need to try everything if you learn intelligently from what you’ve already tried. Build a model of the space. Balance what you know with what you don’t. Choose your next experiment to maximise what you’ll gain from it.

That principle — careful, informed exploration of an uncertain space — is one of the most useful ideas machine learning has to offer. It just happens to also produce a very good cake.


The Recipe: Try It At Home

For the record: after fifty iterations of Bayesian Optimisation across a five-dimensional ingredient space, the system converged on a recipe that scored within 3% of theoretical perfection across all five objectives simultaneously.

Those optimised ratios — [0.375, 0.243, 0.232, 0.196, 0.188] — went into the Claude API. What came out was this.


AI-OPTIMISED CAKE RECIPE — GORDON RAMSAY STYLE

Right, listen up! Some clever AI has crunched the numbers on the perfect cake ratios, and I’m going to show you how to execute this properly. No shortcuts, no bloody mistakes — we’re making this cake PERFECT!

INGREDIENTS

  • 450g plain flour (that’s your base — don’t you dare use self-raising!)
  • 290g caster sugar (not granulated — CASTER!)
  • 280g unsalted butter, room temperature (if it’s cold, you’ve already failed!)
  • 235ml whole eggs — about 4–5 large eggs, beaten
  • 225ml whole milk, room temperature
  • 2 tsp baking powder (fresh — not that stale rubbish from 2019!)
  • 1 tsp fine sea salt
  • 2 tsp pure vanilla extract (not imitation — I’ll know!)

METHOD

  1. Preheat your oven to 180°C / 350°F. Grease and flour that 9-inch pan properly — every bloody corner.
  2. Cream the butter and sugar for 4–5 minutes until pale and fluffy. I want to see volume. If it looks flat, you’re not beating it enough.
  3. Add eggs gradually — one at a time. Rush this and it’ll curdle faster than a bad relationship. Beat well after each addition.
  4. Sift the flour, baking powder and salt together. SIFT IT. We’re not making concrete here.
  5. Alternate the dry ingredients and milk in three additions: flour, milk, flour, milk, flour. Mix until just combined. Overmix and you’ll have a tough, chewy disaster.
  6. Fold in vanilla with confidence. Don’t overwork it.
  7. Pour into the prepared pan and level properly with an offset spatula.
  8. Bake for 28–32 minutes. Test with a skewer — it should come out with just a few moist crumbs. Dry cake is unforgivable.
  9. Cool in the pan for 10 minutes, then turn out onto a wire rack.

CRITICAL POINTS

  • Room temperature ingredients are non-negotiable
  • Don’t open the oven door for the first 25 minutes or it’ll collapse
  • This AI got us precise ratios — respect them

Get it right, and you’ll have a cake worthy of service. Mess it up, and you’ll be starting over.

Beautiful.


The mathematics gave us the ratios. Gordon gave us the recipe. Between them, something genuinely useful came out — and that, as much as anything, is the point of this whole experiment.


The Proof Is in the Pudding

So I baked it. Exact recipe, exact measurements, no shortcuts — Gordon would expect nothing less.

The verdict? Mixed. In the best possible way.

The mixture was perfect. Texture, consistency, the way it came together in the bowl — the optimised ratios delivered exactly what the model predicted. Five numbers between zero and one, translated into real ingredients, and the mathematics held up in a real kitchen. That part worked.

The baking time didn’t. Claude said 28–32 minutes. The cake needed 45.

I caught it — checked, tested with the skewer, added time — but it was the first signal that the experiment wasn’t quite finished. Following the recipe to the letter, without judgement, would have produced something underbaked and disappointing.

And that led to the more interesting discovery: the recipe needed more than correcting. It needed refining. The original prompt asked Claude for a cake recipe and got one — technically correct, but generic. A single 9-inch layer, no context, no intended outcome. So I went back and iterated.

Specify two 8-inch tins and you get a Victoria sponge structure. Specify the occasion, the texture, the finish, and the recipe shifts accordingly. Each prompt refinement produced a more useful, more precise output. The AI wasn’t wrong — it was answering the question it had been asked. Better questions produced better answers.

This is prompt engineering: the practice of shaping what you ask an AI in order to shape what it gives back. It’s not a workaround. It’s a skill — and in this experiment, it turned out to be the third essential ingredient.

Which means the original equation needs updating:

ML + GenAI + Prompt Engineering = Cake

The machine learning optimised the ingredient ratios. The generative AI translated them into a human-readable recipe. Prompt engineering refined that recipe into something genuinely usable. And a human with an oven, a skewer and some accumulated baking knowledge held the whole thing together.

That’s not a story about AI falling short. It’s a story about what genuine human-AI partnership looks like in practice. Each part of the system — the optimiser, the language model, the prompt, the person — contributed something the others couldn’t. None of it worked without the others.

The cake, for the record, was excellent.

Gordon would probably still find something to complain about.

Garbage In, Expensive Garbage Out: Why Dirty Data Breaks Your AI

Diagram illustrating a data quality cleansing and governance engine with input elements like contaminated feedstock, bias, and outliers, leading to AI-ready data and dependable outcomes.

We’ve all heard “garbage in, garbage out.” With AI, it’s worse: garbage in, expensive garbage out.

In earlier posts in this series, we described data as the crude oil of the modern enterprise — raw, unrefined, and only valuable once it’s been properly processed. We walked through the refinery: the pipelines, the engineers, the transformation stages that turn raw data into something a model can actually learn from.

This post is about what happens when the refinery fails. When contaminated feedstock gets through. When the fuel that reaches your AI engine is dirty — and the engine runs anyway, at scale, with complete confidence, in entirely the wrong direction.

Because that’s the thing about modern AI. It doesn’t grind to a halt when the data is bad. It doesn’t throw a warning light. It just learns whatever patterns you give it, optimises for whatever labels you’ve defined, and delivers outputs with impressive precision — even when those outputs are wrong.

Bad data doesn’t give you cheap automation. It gives you expensive, automated mistakes.


What “Data Cleansing” Really Means

“Data cleansing” sounds like housekeeping. Tidy up a few rows, fix a column type, deduplicate some records. Job done.

In practice, it’s something more consequential than that. Every cleansing decision is a decision about what version of reality your AI will learn.

Back at the refinery, crude oil goes through a series of processes before it becomes usable fuel — each stage removing a different class of impurity, each stage making deliberate choices about what to keep and what to discard. Data cleansing works the same way.

At a minimum it involves fixing obvious errors: swapped fields, broken timestamps, invalid encodings. It means standardising how the same thing is represented — UK and U.K. and United Kingdom are the same country, but a model seeing them as three separate values will treat them as three separate signals. It means aligning schemas and units across systems so that a model can learn genuine patterns rather than format quirks. And it means making careful decisions about which attributes actually drive the outcome you care about, and which ones just add noise.

Here’s what makes this harder than it looks: many of those decisions involve assumptions that are invisible once they’re baked in. Which rows are outliers — and which are rare but important edge cases the business genuinely needs to handle? Which labels represent ground truth — and which carry the fingerprints of historical bias? Which data sources are trustworthy enough to be treated as signal?

In traditional reporting, a bad assumption might skew a dashboard. In AI, it gets learned, encoded, and scaled out. The model will faithfully reconstruct and amplify whatever patterns you present as truth.


How Dirty Data Shows Up

Dirty data is rarely one dramatic error. It’s a collection of small issues that quietly pull the model off course — individually manageable, collectively damaging. Here are the usual suspects.

Biased data is perhaps the most insidious. It occurs when the data doesn’t represent the population or behaviour you’re actually trying to serve. Historical decisions — approvals, pricing, hiring — get recorded as objective truth, even when they weren’t. Certain geographies, demographics or product lines are over-represented. Training sets include only “successful” cases and ignore failures or edge cases. The model faithfully learns yesterday’s prejudices and scales them out. Performance looks excellent on the same skewed distribution; it falls apart the moment you try to broaden usage.

Outliers and anomalies are trickier, because they’re not always wrong. One-off promotions, emergency discounts, crisis events, logging glitches, batch replays — these all produce data points that sit outside the normal pattern. Handle them badly and models overfit to noise. Ignore them entirely and you may have scrubbed away exactly the edge cases the business needs to handle — the fraud signals, the safety events, the compliance-relevant exceptions.

Dubious or unverified sources are a growing problem as data estates expand. Scraped web data with no curation. Third-party feeds glued into production pipelines with minimal validation. Synthetic data mixed with real data and never labelled as such. The model learns correlations that only exist in that one noisy dataset, inherits someone else’s labelling mistakes as ground truth, and produces outputs that look confident and are quietly wrong.

Nulls, blanks and missingness look simple but aren’t. A missing value in one column might mean the data was never collected, or the user chose not to answer, or the field was redacted for legal reasons, or a legacy system simply didn’t capture it. Treating all of those the same — filling everything with zero or a global mean — creates fake patterns. Models confuse “unknown” with “none”, and segments with more missing data become systematically under or over-served as a result.

Inaccurate or stale data is the fuel that’s gone off. Records without proper timestamps or versioning. Manual entry errors and mis-keyed IDs. Reference data that was accurate two years ago and hasn’t been updated since. Models trained on stale data optimise for a world that no longer exists — old pricing, old customer behaviour, old market conditions — and deliver what look like strong offline metrics that evaporate in production.

Inconsistent definitions are often the hardest to catch because the data looks clean. “Customer” means the billing account in one system, the individual in another, the household in a third. “Churn” means no login for thirty days in one team’s definition, no revenue in ninety days in another’s. The model trains on a Frankenstein label that no one fully agrees on, produces outputs with impressive AUC scores, and is quietly solving the wrong problem.


Why AI Gets It Wrong Even When the Model Looks Fine

Most AI failures aren’t about the algorithm. They’re about the data pipeline.

Dirty data corrupts the learning process in ways that don’t show up until the model is in production. If your labels are biased or conceptually wrong, your model is optimising for the wrong behaviour — and it will do so very efficiently. If the training and validation data share the same quirks (the same logging artefacts, the same process shortcuts, the same dummy values), the model will look excellent right up until it meets real-world data that doesn’t carry those quirks.

Modern models are extraordinarily good at interpolating through messy data. That’s part of what makes them powerful. It’s also what makes dirty data so dangerous. You can get good-looking metrics on a flawed test set, a slick demo that works beautifully on carefully prepared examples, and a deployment that quietly fails in the wild. The model isn’t broken. It’s just running on contaminated fuel — confidently, at scale, in the wrong direction.


Timing Is Part of Data Quality

We tend to talk about data quality as if it were purely about correctness. In AI systems, freshness is just as important.

If you’re making real-time decisions on yesterday’s snapshot, your fraud models are lagging the attackers, your recommendation models are pushing last week’s interests, and your operational models are optimising for a backlog that’s already changed.

But the deeper problem is concept drift. The world changes — new products, new pricing, new channels, macro events that reshape customer behaviour entirely. If your training data doesn’t keep up, the model keeps extrapolating from patterns that no longer hold, with no way to know that the ground has shifted beneath it.

And then there are feedback loops. Many AI systems are closed loops: the model makes decisions, those decisions influence what data gets collected next, and that data is used to retrain the model. If the early data is dirty or biased, the loop amplifies the problem. The model stops seeing counter-examples. Segments that are mis-served stay mis-served. The system drifts quietly into a self-reinforcing corner of the feature space.

Data quality, properly understood, is correctness multiplied by relevance multiplied by timeliness. If any of those three factors drops to zero, the output is still expensive garbage — however impressive the model architecture.


Smarter Cleansing, Not Aggressive Pruning

Given all of this, it’s tempting to treat cleansing as “delete anything that looks messy.” That’s the wrong instinct — and it can be as damaging as doing nothing.

You don’t want to scrub away minority groups because they represent small sample sizes. You don’t want to discard edge cases that reveal failure modes you need to handle. You don’t want to erase records of bad historical outcomes that show you exactly where bias or process issues lived.

Smarter cleansing means diagnosing and documenting bias rather than pretending it isn’t there — quantifying where the data is skewed, and deciding consciously whether to re-weight, augment or treat those segments separately. It means treating outliers as signals rather than inconveniences, and recognising that for some use cases — fraud, anomaly detection, safety — the outliers are the whole point. It means making missingness explicit, using indicator variables or separate categories rather than silently filling gaps with a magic constant. And it means tracking lineage and provenance so that when something goes wrong in production, you can debug the pipeline — not the model weights.

Cleansing isn’t housekeeping. It’s data governance. And it’s the most important thing you can do for AI that’s actually trustworthy.


Keeping the Refinery Running: Where Dell Comes In

All of this is straightforward in principle and genuinely hard in practice — especially when your data estate is spread across on-premises infrastructure, cloud environments and edge locations, and you’re working with structured tables alongside documents, images, logs and sensor data simultaneously.

The challenge isn’t understanding what clean data looks like. It’s building the refinery infrastructure that produces it consistently, at scale, without burying your data engineers in manual work.

That’s the problem Dell is focused on solving.

It starts before the pipelines. Dell’s Data Strategy Services work with data engineers, architects and business stakeholders to define what “good data” actually means for each AI use case — not generically, but specifically: which sources are trustworthy, where bias and gaps exist, what quality and governance requirements each workload demands, and what the end-to-end data management roadmap needs to look like to get there. Engineers build better pipelines when they have a clear, agreed target for the fuel they’re supposed to be producing.

The foundation underneath those pipelines is the Dell AI Data Platform — a unified, modular platform that stores and manages structured, semi-structured and unstructured data through storage engines like PowerScale and ObjectScale, and uses Data Engines to clean, organise and optimise information from applications, devices and other sources. Built-in security and governance — access controls, data masking, encryption and cyber-resilient features — mean that sensitive data stays protected without creating friction for the teams who need to work with it.

On top of that foundation, a significant portion of what data engineers spend their time on — deduplication, format standardisation, missing data imputation, anonymisation — can be automated. Dell’s platform combines AI-driven preprocessing with automated cleansing at scale, so engineers aren’t manually patching pipelines for every new data source. The result is clean, consistently structured data flowing through to models, rather than a patchwork of workarounds that looks fine until it doesn’t.

But clean storage and cleansing aren’t enough on their own. AI isn’t a single batch job — it’s a continuous lifecycle: data preparation, validation, model training, deployment, monitoring, retraining, and back again. The Dell Data Orchestration Engine acts as the control plane for that entire loop, connecting ingestion, pipeline automation, validation, model triggers and governance enforcement across structured and unstructured, batch and streaming data. Critically, it enforces data quality gates before models ever see the data — so bias, outliers and missingness don’t silently migrate into production systems while no one is watching.

And wrapping all of it is the expertise to make it work in practice. Dell’s data engineering services help teams design and implement pipelines that tag, cleanse, label and anonymise data to produce AI-ready datasets in the right format at the right time — all as part of the broader Dell AI Factory with NVIDIA, a vertically integrated stack of storage, data engines, orchestration, GPUs and services built for the next decade of AI workloads.


Expensive Models, or Dependable Outcomes?

Powerful models make it easy to forget that data is the real engine.

You can invest heavily in GPUs, premium model architectures and sophisticated orchestration. But if the fuel going into those systems is dirty — late, inconsistent, biased, poorly understood — you won’t get cheap insight. You’ll get expensive, high-confidence mistakes, delivered at scale, by a system that has no idea anything is wrong.

The organisations that win with AI won’t simply have the biggest models. They’ll have the cleanest, best-understood data, flowing through governed and automated pipelines, built by data engineers who are spending their time on value rather than firefighting.

That’s the shift the Dell AI Data Platform is designed to support: from garbage in, expensive garbage out — to clean data in, dependable outcomes out. The refinery has to work. Everything downstream depends on it.

If you’d like to explore this further, Dell has gone deeper on both the data-quality story and the platform behind it:
– Keep Your AI Engine Running with Good, Clean Data
– Dell AI Data Platform