by 
My secondary school maths teacher, Miss Smith couldn’t be messed with. She wasn’t a big woman, and she didn’t raise her voice, but she had a way that the other teachers lacked — a quiet certainty that made you sit up straighter without quite knowing why. To me, maths was never really about numbers. It was puzzles. Logical puzzles, written in a language of numbers, functions and variables, waiting to be solved. Get the language right, and the answer had to follow. There was no arguing with it, no opinion to have. It simply was. It was then I formed the idea everything can be described in numbers. The “code” of the universe. The basis of AI.
That same love of numbers and logic — the idea that a well-posed problem contains its own answer, if you can only speak its language — is the thread that runs from those classroom puzzles all the way to the earliest machine learning algorithms, some of which trace their roots back centuries. Linear regression. Bayes’ theorem. Decision trees. K-Nearest Neighbour. And more besides. These aren’t relics. They sit quietly inside a huge proportion of the Python solutions doing real work in the world today, and they still deserve their place at the table. If you want to predict next quarter’s revenue from ten years of clean, structured data, you very rarely need anything more exotic than a well-fitted regression. Boring AI wins every time.
But somewhere in the twentieth century, ambition outgrew the puzzle-solver. Statistical models are, at their core, an exercise in description — here is the shape of the data, here is the line that best fits it, here is the probability given what we’ve already observed. What researchers wanted next was something different in kind, not just in degree: a machine that didn’t just describe patterns in data, but learned the way a brain does. Not a better equation. A different kind of thing entirely.
In this series I want to explore what I’m calling Neo-ML — neo machine learning — and how it has evolved to give us deep learning, computer vision, generative AI, and the ability to talk to our data and have it talk back in natural language. We’ll move from the artificial neuron through feedforward networks, into convolutional networks and computer vision, on to recurrent networks and long short-term memory, and finish at autoencoders and the doorway to generative AI. I’ve just coined a new term — Neo-ML — and by the end of this series you’ll understand exactly why it deserves one.
Let’s start this series with the round dot you see in most deep learning diagrams – the node, or neuron. And it’s just a mathematical equation in software. A function.
Part 1: The Artificial Neuron (Node)
The Biological Inspiration
The story starts, as so many things in computing do, with a question that sounds almost philosophical: can we build a machine that thinks?
In 1943, a neurologist named Warren McCulloch and a mathematician named Walter Pitts sat down and asked a more tractable version of that question. They weren’t trying to replicate consciousness. They were trying to model the simplest possible unit of biological computation — the neuron — in mathematical terms. Their paper, A Logical Calculus of the Ideas Immanent in Nervous Activity, described a neuron as something that receives signals, adds them up, and fires if the total crosses a threshold. Binary. On or off. Yes or no.
It was a brutal simplification of what a real neuron does. But it was the right simplification. It captured the essential logic: inputs come in, something happens in the middle, and a decision comes out.
That structure — inputs, transformation, output — is the skeleton of every artificial neuron ever built. Everything else in this series is an elaboration of it.
What an Artificial Neuron Actually Does
Let’s make it concrete.
Imagine you’re trying to decide whether to take an umbrella when you leave the house. You look at a few things: the sky looks grey, the forecast says 70% chance of rain, and you remember it rained yesterday. Each of those is an input. But they don’t all carry the same weight. The forecast matters more than the colour of the sky. Your memory of yesterday matters least of all.
An artificial neuron works the same way. It takes a set of inputs — numbers, each representing some feature of the world — and multiplies each one by a weight. The weight is the neuron’s way of saying how much it cares about that particular input. It then adds all those weighted inputs together, throws in a small constant called a bias, which lets it shift its behaviour even when all inputs are zero, and passes the result through a function that decides what to output.
In plain terms: the output equals an activation function applied to the weighted sum of the inputs, plus a bias. That last piece — the activation function — is where things get interesting.
The Activation Function: Where Non-Linearity Lives
If you removed the activation function and just kept the weighted sum, you’d have a linear model. Useful, but limited. The whole point of neural networks is that the real world is not linear. The relationship between the pixels in a photograph and the word “cat” is not a straight line. Neither is the relationship between a sentence and its sentiment, or between a patient’s symptoms and their diagnosis.
The activation function introduces non-linearity. It takes the raw weighted sum and squashes it, bends it, or clips it into a shape that lets the network learn curved, complex relationships.
The earliest activation function was the step function — exactly what McCulloch and Pitts described. If the sum crosses the threshold, output 1. Otherwise, output 0. Clean, but not differentiable, which matters enormously when you want to train the network (more on that in the next post).
The sigmoid function came next. It takes any input and maps it smoothly to a value between 0 and 1, producing an S-shaped curve. For a long time, it was the default choice. It has problems at the extremes — the gradients become vanishingly small, which makes learning slow — but it was a significant step forward.
Close behind sigmoid came the hyperbolic tangent, or tanh. It solves a similar problem in a similar shape — another smooth S-curve — but where sigmoid squashes everything into a range between 0 and 1, tanh maps to a range between -1 and 1. That zero-centring matters more than it sounds like it should: because tanh’s outputs can be negative, the weights in the next layer get pushed both up and down during training rather than only ever nudged in one direction. In practice, that tends to make networks converge faster than sigmoid does. It doesn’t solve the vanishing gradient problem — the curve still flattens out at the extremes, just like sigmoid’s does — but for a long stretch of the field’s history, tanh was the safer default of the two.
Today, the most widely used activation function is something almost embarrassingly simple: the Rectified Linear Unit, or ReLU. If the input is negative, output zero. If it’s positive, output the input unchanged. That’s it. No smooth curve, no elegant formula. Just a hinge. And yet ReLU, and its variants, power the majority of modern deep learning systems. Sometimes the right answer really is the simple one.
Why This Is a Different Kind of Thing
Here is the point that the name Neo-ML is trying to capture.
Classical machine learning algorithms — regression, decision trees, support vector machines — are built around explicit mathematical relationships. You, the engineer, decide what features matter. You decide how they relate to the output. The algorithm finds the best parameters to fit those relationships to the data. The structure of the model is yours. The algorithm fills in the numbers.
An artificial neuron — and the networks built from them — works differently. You don’t specify the relationships. You specify the architecture: how many neurons, how they’re connected, what activation functions they use. Then you show the network data, and it learns the relationships itself. The weights and biases are not chosen by you. They are discovered, through a process of trial and error at a scale no human could manage, by the training algorithm.
This is not a quantitative difference. It is a qualitative one. The model is not a description of the world that you wrote. It is a description of the world that the machine wrote, in a language of numbers that no human fully reads.
That shift — from models we design to models that design themselves — is what makes neural networks a different kind of thing. Not smarter than classical ML in every situation. Not always the right tool. But categorically different in what they are and how they work.
The Perceptron: The First Learnable Neuron
McCulloch and Pitts gave us the structure. But their neuron couldn’t learn. The weights were set by hand. It took another fifteen years, and a psychologist named Frank Rosenblatt, to build the first neuron that could update its own weights based on experience.
In 1958, Rosenblatt introduced the Perceptron. The idea was simple: show the neuron a labelled example, let it make a prediction, compare the prediction to the correct answer, and nudge the weights in the direction that would have made the prediction better. Do this thousands of times, across thousands of examples, and the weights gradually converge on something useful.
The Perceptron learning rule is the ancestor of every training algorithm in modern deep learning. Backpropagation, gradient descent, Adam optimisers — all of them are elaborations of the same core idea: measure the error, and adjust the weights to reduce it.
Rosenblatt was famously optimistic about what the Perceptron could do. Too optimistic, as it turned out. In 1969, Marvin Minsky and Seymour Papert published Perceptrons, a rigorous mathematical analysis that showed a single-layer Perceptron could not solve problems that weren’t linearly separable. The classic example is XOR — a logical operation so simple a child can understand it, but one that a single Perceptron cannot learn. The field went quiet for a decade. The first AI winter had arrived.
The solution, when it came, was obvious in retrospect: stack the neurons. Put one layer’s outputs into the next layer’s inputs. Build depth. That is the feedforward neural network, and it is where we go next.
What We’ve Established
A single artificial neuron is a small, precise thing. It takes inputs, weights them, sums them, applies a non-linear function, and produces an output. It can learn — in the sense that its weights can be adjusted to improve its predictions. And it is, in a meaningful sense, a different kind of computational object from the statistical models that came before it.
One neuron, on its own, is not very impressive. It can draw a line through data. It can make a binary decision. It is, in the end, not so different from logistic regression.
But connect enough of them together, in the right way, and something remarkable happens. The network stops drawing lines and starts drawing boundaries of arbitrary complexity. It stops recognising simple patterns and starts recognising faces, voices, sentences, intentions.
That emergence — the way complexity arises from simplicity, the way the whole becomes vastly more than the sum of its parts — is what the rest of this series is about.
Miss Smith would have approved. It turns out the universe is still a puzzle. We’ve just found a new way to speak its language.
1 Comment