by Transparency and Explainability – inside the black magic box
Ever since I can remember, I’ve wanted to know how things work. Not just that they work — but why, and what’s going on inside.
My parents had a name for it: Fiddle Fingers. Clocks, radios, household appliances — nothing was safe. I’d take them apart with genuine curiosity and varying degrees of success. The parts I couldn’t reassemble quietly disappeared under my bed. I even unscrewed the back of a plug once — purely to see electricity. The thunderbolts that shot up my arm was, in hindsight, a reasonable price for the lesson. My parents, to their credit, were more understanding than the appliances deserved.
Later I studied motor vehicle engineering — which at least meant I could finally take things apart professionally. And now, working through my MSc, I’m still doing the same thing. Still looking under the hood. Still asking what’s in the box.
Which makes the subject of this post a personal one. Because one of the most troubling things about modern AI systems is that they often don’t let you look inside. Not because the technology prevents it — but because transparency and explainability haven’t been treated as priorities. The box stays closed. And when the box is closed, accountability becomes very difficult to defend.
In the previous parts of this series we’ve looked at bias, fairness and accountability — the ethical challenges that emerge when AI systems make decisions that affect people’s lives. This instalment moves into territory that sits underneath all of those: if you can’t see how a system works, and can’t explain what it decided, the other ethical principles become very difficult to uphold.
Transparency and explainability are the mechanisms that make accountability possible. Without them, everything else is aspiration.
Two related concepts, one shared purpose
The terms transparency and explainability are often used interchangeably. They shouldn’t be — they address different things, and the distinction matters.
Transparency concerns the visibility of the system as a whole. How was it built? What data was it trained on? What modelling choices were made, and why? What does the performance data actually show? Transparency enables external scrutiny. It supports governance, auditability and regulatory oversight. Without it, independent evaluation becomes impossible — you’re simply asked to trust the outcome without any means of checking it.
Explainability concerns the individual decision. Not the system in aggregate, but this specific output: why did the model produce this result for this person, in this context, at this moment? In high-stakes settings — healthcare, criminal justice, financial services — that question isn’t academic. It’s a matter of rights.
Think of it this way. Transparency lets you audit the factory. Explainability lets you understand why one particular product came off the line the way it did.
Both matter. And in most real-world deployments of AI today, both are harder to achieve than the marketing suggests.
The three questions explainability has to answer
When we talk about making an AI system explainable, we’re really asking three distinct questions — and each requires a different kind of answer.
The first is about data. What information was used to train the model, and why was it chosen? This isn’t just a technical question. Training data encodes assumptions about the world, and those assumptions shape every output the model produces. If the data can’t be explained and justified, the decisions downstream can’t be either.
The second is about predictions. What features and weights drove this particular output? Why did the model score this applicant lower than another? Which variables carried the most influence, and in what direction? This is where post hoc explanation techniques — tools that interpret model behaviour after the fact — do most of their work.
The third is about the algorithm itself. What are the layers, the thresholds, the decision boundaries? How does the model move from input to output? For simpler models, this question has a direct answer. For more complex ones, it often doesn’t — which is where the central tension of this topic lives.
COMPAS: when a black box meets a courtroom
No case study illustrates the stakes of transparency and explainability more starkly than COMPAS — the Correctional Offender Management Profiling for Alternative Sanctions tool, widely used in the United States to assess the risk that a defendant will reoffend.
Judges used COMPAS scores to inform decisions about bail, sentencing and parole. The scores carried real weight in outcomes that determined whether people went home or went to prison. And yet the algorithm that produced those scores was proprietary. Defendants had no means of understanding how their score was calculated, no ability to identify errors in the underlying data, and no realistic way to challenge the output in court.
In 2016, ProPublica published an investigation showing that COMPAS assigned significantly higher reoffending risk scores to Black defendants than to white defendants with comparable profiles. The tool wasn’t just opaque — it was producing outcomes that were racially skewed in one of the highest-stakes contexts imaginable.
The Loomis v. Wisconsin case reached the Wisconsin Supreme Court, where the defendant argued that using a proprietary, unexplainable algorithm in sentencing violated his right to due process. The court upheld the use of the tool. The algorithm remained a black box.
COMPAS sits at the intersection of everything that matters in this conversation. Transparency was absent — no visibility into the model’s design, data or validation. Explainability was absent — no way to interrogate individual decisions. And the consequences were borne by people who had no recourse and no means of understanding why.
The tension that doesn’t go away
Here is the dilemma that transparency and explainability force us to confront — and it doesn’t have a clean resolution.
The models that tend to perform best on complex, real-world prediction tasks are also the least interpretable. Deep neural networks, gradient boosting models, large ensemble methods — these approaches can achieve superior predictive accuracy precisely because they capture subtle, non-linear relationships in data that simpler models miss. But that complexity comes at a cost: the internal workings become difficult, sometimes impossible, to explain in terms a human can meaningfully interpret.
Simpler models — linear regression, decision trees, rule-based systems — offer genuine interpretability. You can follow the logic from input to output, identify which variables matter and by how much, and explain a decision to the person it affects. But they often sacrifice accuracy to do it. In a noisy, high-dimensional real world, simpler models sometimes just get more things wrong.
This is not a technical problem waiting for a technical solution. It is a genuine ethical trade-off. In some contexts — say, a recommendation engine for a streaming service — that trade-off sits comfortably on the side of performance. In others — a credit decision, a medical diagnosis, a criminal risk score — the question of what we’re willing to sacrifice for accuracy becomes a question of values, not engineering.
Regulatory frameworks are beginning to codify where that line falls. The EU AI Act classifies high-risk AI applications and mandates transparency and explainability requirements accordingly. The GDPR enshrines a right to explanation for automated decisions. But regulation sets a floor, not a ceiling — and the honest truth is that many organisations are still well below it.
What good looks like in practice
Transparency and explainability aren’t binary. They exist on a spectrum, and the appropriate level depends on context — the stakes involved, the people affected, and the regulatory environment in play.
For high-risk applications, the baseline should include clear documentation of training data, modelling choices and performance metrics across demographic groups; post hoc explanation tools that can surface the key drivers of individual decisions; human review mechanisms for decisions that significantly affect individuals; and the ability to audit the system independently — not just internally.
For lower-risk applications, lighter-touch approaches may be proportionate. But the principle remains: the system should be able to account for itself, and the people it affects should have a meaningful way to understand and, where necessary, challenge its outputs.
The temptation to treat explainability as a presentation problem — a dashboard, a label, a percentage confidence score — should be resisted. A number on a screen is not an explanation. An explanation is something a person can interrogate, reason about and act on.
Closing thought
There is a version of AI development where transparency and explainability are treated as compliance tasks — boxes to tick, documentation to file, a report to produce before launch. That version produces systems that look accountable without being accountable.
The harder version asks the question earlier: before a model is selected, before a dataset is assembled, before a use case is approved. It treats interpretability as a design constraint, not an afterthought. It asks whether a complex model is actually necessary, or whether a simpler, more explainable one would serve the purpose well enough.
That version is also the honest version. Because when a system makes a decision that changes someone’s life — and they ask why — “the algorithm is proprietary” is not an answer any ethical organisation should be comfortable giving.
Transparency and explainability have to be built in, not bolted on.
Leave a Reply