by 
Social Impact
Years ago — life before iTunes, before Spotify, before algorithms knew what you liked before you did — I heard the most beautiful song on the radio.
I rushed for a pen and paper. Too late. The DJ had already moved on. I had no title, no artist, no way to find it. Just the music, lodged somewhere in memory, with nowhere to go.
That song stayed with me for years.
Then one day I heard it again. I recognised it immediately — the same ache, the same sound. This time I was ready. Moments later I was on Amazon, searching for what I’d finally caught: Nick Drake. River Man. I bought the CD — Five Leaves Left — and discovered one of the most quietly extraordinary artists I have ever heard.
Nick Drake never had a hit in his lifetime. He sold a few thousand records. He died at twenty-six, largely unknown. His reputation grew slowly, entirely through human recommendation — one person telling another, a song surfacing unexpectedly on a radio programme, a stranger pointing someone in the right direction. Decades after his death, he is considered a towering influence.
I think about that story when I think about what recommendation algorithms do — and what they can’t do.
Those moments of genuine discovery are becoming rarer. And it is not an accident.
In previous parts of this series we have examined bias and fairness, privacy and consent, and transparency and explainability. Each of those topics asks what happens when AI gets something wrong in the moment — a biased decision, a privacy violation, an unexplained outcome. Social impact asks a different and harder question: what happens when AI gets things right, consistently, at scale — and the cumulative effect is still harmful?
This is the part of AI ethics that is easiest to overlook. There is no single decision to challenge. No obvious moment of failure. Just a system doing exactly what it was designed to do, and a world quietly changing around it.
The Feedback Loop
Machine learning systems influence social structures when deployed at scale. Decisions about credit approval, employment screening, content recommendation, and public resource allocation affect opportunities and outcomes for individuals and communities — not just once, but continuously, and often invisibly.
The mechanism behind many social harms is the feedback loop: a system trained on past behaviour makes decisions that shape future behaviour, which then becomes the training data for the next version of the model. Each cycle reinforces what came before. Small biases become structural ones. Initial disparities widen. And because every individual decision appears reasonable, the cumulative drift goes unnoticed until the damage is done.
Example One: The Playlist That Narrows
Consider a music streaming platform that recommends songs based on what users have previously listened to. A user starts with a few popular mainstream artists. The system, doing its job, recommends more of the same. Over time, the user is repeatedly exposed to the same genres, the same sounds, the same familiar names — while niche and emerging artists remain invisible.
The feedback loop runs like this: past listening shapes recommendations, recommendations reinforce listening patterns, and those patterns feed the next round of recommendations. Popular artists become more popular. Smaller artists remain underrepresented. Not because the algorithm intended to marginalise them — but because it was optimised for engagement, and engagement follows familiarity.
Each recommendation, taken alone, is perfectly reasonable. A user who likes one thing probably likes similar things. But the cumulative effect reshapes what people discover, what gains cultural traction, and ultimately who earns a living from their music. A technical optimisation becomes a cultural force. And nobody pressed a button that said “narrow the culture.”
Example Two: The Loan That Was Never Offered
Now consider a model used to decide who gets approved for a loan.
If the historical data that trained the model reflects decades of biased lending practices — and it often does — the model will learn to reject applicants from certain demographic groups at higher rates. It is not making a racist decision in the way a human might. It is making a statistically grounded one, based on patterns in the data. But those patterns are the residue of past discrimination.
The feedback loop here is more severe, and the stakes are higher:
- Fewer approved loans → fewer opportunities to build credit, start businesses, or buy homes
- Fewer opportunities → continued financial disadvantage
- Continued disadvantage → future data that confirms the model’s original assessment
The system appears statistically accurate. It is. And it is also socially harmful. The two things are not mutually exclusive — which is what makes this so difficult to resolve by purely technical means.
Scale Changes Everything
A small systematic error, repeated at scale, produces significant societal consequences. This is the central insight of social impact analysis in AI.
A single biased loan decision is a wrong that can be appealed. A biased model making ten thousand decisions a day, over years, without review, is a structural shift in who gets access to capital. A streaming algorithm that slightly deprioritises independent artists across a platform of three hundred million users does not just affect listening habits — it shapes the economics of an entire industry.
This is why social impact analysis requires evaluating not only individual predictions but cumulative effects. It requires monitoring mechanisms capable of detecting harm early — before it becomes entrenched. It requires stakeholder engagement with affected communities, because the people most likely to identify risks are often the ones the system is making decisions about. Technical analysis, however rigorous, cannot see what it has not been designed to look for.
Machine learning systems are not neutral tools. They are components of socio-technical systems — embedded in institutions, shaped by history, and capable of reinforcing or redirecting the structures they operate within. Their evaluation must extend beyond statistical metrics to include institutional and societal considerations. That is not a soft requirement. It is an engineering one.
Asking “does the model perform well?” is no longer sufficient. The question that matters is: “What does the world look like after this model has been running for five years?”
Social impact has to be built in, not bolted on.
Next: Part 6 — Ethical Trade-offs. The honest conclusion: there are no perfect answers. Only deliberate choices.
Leave a Reply