Artificial Neural Network (ANN) by example

As kids growing up in the 70s with no internet and no video games, we explored the world around us. Our curiosity fueled adventure and discovery. For me, I needed to understand how things worked. I remember one day in my bedroom having the thought, “I wonder what electricity looks like?” So I took the back off a plug and plugged it into the wall socket. I didn’t get to see electricity, but I certainly got to feel it. My right arm jolted as 240 volts held me pinned to the wall plug. Finally, with some effort, I grabbed my right arm with my left and pulled myself away. Of course, I never told my parents.

Growing up in an age of childhood freedom gave me curiosity. And that same curiosity is why this blog exists. I’m not going to walk through a neural network diagram with dots and lines. In this blog. Instead I’ll walk through the code of a simple ANN. Don’t worry, it’s not shocking.

In the last two posts we built up the ideas. A single artificial neuron: weighted inputs, a bias, an activation function, a decision. Then a network of them: layers stacked so that each one hands its output to the next, learning boundaries no single neuron could draw alone. We talked about backpropagation, loss functions, and why depth changes what’s possible.

Now let’s get under the hood.

The code below is a real, working neural network — not a toy illustration, not pseudocode. It’s written in Python using TensorFlow and Keras, which are the tools most practitioners actually reach for. The problem it solves is deliberately simple: given a person’s age, state, income, and political leaning, predict whether they work in Engineering or Education. Two possible answers. One network to find the boundary between them.

That’s a binary classification problem — the same category as “will this machine fail in the next 24 hours?” from the factory floor. Same structure, different data.

Step 1: The Data

Two hundred people for training, forty held back for testing. Each row is one person. The Sector column is the answer — 1 for Engineering, 0 for Education. Everything else is a feature: the inputs the network will learn from.

Look at the numbers in the data and you’ll notice something. Age isn’t stored as 32 or 45 — it’s stored as 0.32 or 0.45. Income isn’t 82,000 — it’s 0.82. Everything has been scaled down onto a comparable footing, roughly between 0 and 1. This is data normalisation, and it matters more than it sounds. Without it, the raw income figure — a number in the tens of thousands — would dominate every weighted sum in the network simply by being numerically large, regardless of whether income is actually the most informative feature. Data normalisation levels the playing field so the network can decide for itself what matters.

The state and political leaning columns work differently. A state isn’t a number on a scale — Oregon isn’t “more” than Maine in any meaningful sense. So instead of encoding state as a single column with values 1, 2, 3, each state gets its own binary column: State_OR, State_ME, State_OK. A person from Maine gets a 1 in State_ME and a 0 everywhere else. This is called one-hot encoding, and it’s the standard way of feeding categorical data into a neural network without accidentally implying an order that doesn’t exist.

Step 2: The Architecture

Stop for a second. This is it. This is the neural network. Not a diagram. Not a concept. Five lines of Python code that define the entire structure of an ANN — the layers, the neurons, the activation functions, everything. When people talk about “building a neural network,” this is what they mean. Let’s walk through each line.

Input(shape=(8,)) — the doorway. Eight features come in: age, three state columns, income, three political leaning columns. This layer doesn’t compute anything; it just tells the network what shape to expect.

Dense(16, activation=’relu’) — the first hidden layer. Sixteen neurons, each one connected to all eight inputs. Each neuron takes a weighted sum of those eight values, adds a bias, and passes the result through ReLU. If the result is positive, it passes through unchanged. If it’s negative, it becomes zero. Sixteen different weighted combinations of the same eight inputs, each one learning to notice something slightly different about the data.

Dropout(0.2) — this one doesn’t appear in most diagrams, but it’s doing something important. During training, it randomly switches off 20% of the neurons in the previous layer on each pass. Not the same 20% every time — a different random selection each time. This sounds counterproductive. Why would you deliberately break your own network? Because it forces the network to learn redundant, overlapping representations rather than leaning too heavily on any single neuron. A network that can only function when every neuron is present is a fragile one. Dropout is the training equivalent of practising a skill with one hand tied behind your back so that when both hands are free, you’re stronger for it. At prediction time, Dropout switches off entirely — all neurons active, full strength.

Dense(16, activation=’relu’) — a second hidden layer, identical in structure to the first. Its inputs are the outputs of the first hidden layer, not the raw data. This is where depth earns its keep: the second layer isn’t looking at age and income directly, it’s looking at whatever the first layer learned to notice about age and income. Abstractions built on abstractions.

Dense(1, activation=’sigmoid’) — the output layer. One neuron, because there are two possible answers and one probability is enough to express both. Sigmoid squashes the result to a number between 0 and 1. Above 0.5, the network says Engineering. Below 0.5, it says Education. The closer to the extremes, the more confident the prediction.

The whole network has 433 trainable parameters — 433 individual weights and biases that start as random numbers and will be adjusted, thousands of times, until they mean something. Think about that for a moment. Two hidden layers of 16 neurons each, connecting 8 inputs to a single output, and the entire intelligence of this network — everything it knows about predicting whether someone works in Engineering or Education — lives in 433 numbers. The model summary tells you the memory footprint too: 1.69 KB. Not megabytes. Not gigabytes. One point six nine kilobytes. A text message takes up more space. And yet those 433 numbers, tuned correctly, can make predictions a human couldn’t easily articulate as a rule.

Step 3: Compile — Choosing the Learning Rules

Before training can begin, the network needs two things: a way to measure how wrong it is, and a strategy for getting less wrong.

binary_crossentropy is the loss function. For a binary classification problem — two possible outcomes — this is the standard choice. It measures the gap between what the network predicted and what the correct answer actually was, and it comes down hardest on predictions that were confident and wrong. A network that says “90% sure it’s Engineering” when it’s actually Education gets penalised far more than one that hedged at 55%.

adam is the optimiser — the strategy for adjusting the weights after each error is measured. Adam is an adaptive optimiser, meaning it doesn’t nudge every weight by the same amount. It keeps track of how each weight has been moving and adjusts its step size accordingly. Weights that have been oscillating get smaller nudges; weights that have been consistently moving in one direction get larger ones. In practice, Adam converges faster and more reliably than plain gradient descent for most problems, which is why it’s become the default starting point for almost everyone.

Step 4: Training — Being Wrong, Repeatedly, on Purpose

This is where the learning happens. model.fit runs the training loop.

epochs=50 means the network will see the entire training dataset fifty times. Each pass through the data is one epoch. On each pass, it makes predictions, measures its error with the loss function, and uses backpropagation to work out how every one of those 433 weights contributed to that error. Then Adam nudges each weight in the direction that reduces it.

batch_size=16 means the network doesn’t wait to see all 200 training examples before updating its weights. It processes 16 at a time, updates, processes the next 16, updates again. This is called mini-batch gradient descent, and it’s faster and often more stable than waiting for the full dataset.

validation_split=0.2 holds back 20% of the training data — 40 people — as a validation set. The network never trains on these; they’re used purely to check, after each epoch, whether the network is actually learning something general or just memorising the training data. Watch the training output and you’ll see two accuracy figures per epoch: training accuracy and validation accuracy. If training accuracy climbs while validation accuracy stalls or falls, the network is overfitting — learning the training data by heart rather than learning the underlying pattern.

Here’s what the first few epochs look like:

Epoch 1: the network is barely better than a coin flip. The weights are random, the predictions are noise. By epoch 40, training accuracy has climbed to 71%. The network is learning. The validation accuracy tells a more cautious story — it peaks around 50%, suggesting the network is finding real patterns but the problem is genuinely hard with only 200 training examples. More data would help.

Step 5: Evaluate — Honest Accounting

Test accuracy comes in at 65%. On 40 unseen people, the network gets 26 right. That’s better than chance, but the classification report tells the more interesting story:     

When the network predicts Engineering, it’s right 93% of the time — high precision. But it only catches half the actual engineers — low recall. It’s cautious about calling Engineering, so it misses a lot of them. Education is the mirror image: it catches almost everyone who works in Education (93% recall), but half its Education predictions are wrong (50% precision).

This is exactly the precision-recall trade-off we touched on with the factory floor. Accuracy alone — 65% — doesn’t tell you this. The confusion matrix does:

Thirteen engineers correctly identified. Thirteen engineers incorrectly called Education. One educator incorrectly called Engineering. Thirteen educators correctly identified. The network has a bias toward predicting Education. Understanding why — whether it’s the class balance in the training data, the features available, or the architecture — is where the real diagnostic work begins.

Step 6: Prediction — The Network in Use

The network is trained. Now it earns its keep.

A new person walks in: age 32, from Maine, income $82,000, politically moderate. The code normalises those inputs exactly as the training data was normalised — age becomes 0.32, income becomes 0.82 — and feeds them through the network. The 433 weights, now tuned through 50 epochs of training, do their work. The output neuron returns a single number: 0.7066.

Above 0.5. The network says: Education.

That number — 0.7066 — isn’t a certainty. It’s a probability. The network is saying “I think there’s about a 71% chance this person works in Education.” The threshold of 0.5 is a choice, not a law. In a real application, you might raise it to 0.7 or 0.8 if the cost of a wrong prediction is high. Lower it if missing a true positive is the bigger risk. The threshold is yours to set; the probability is the network’s to produce.

What Just Happened

Six steps. One hundred and something lines of Python. A network that started knowing nothing — random weights, random predictions — and ended up with a view of the world encoded in 433 numbers that no human chose or wrote down.

The architecture was ours: eight inputs, two hidden layers of sixteen neurons each, one output. The activation functions were ours: ReLU in the hidden layers, sigmoid at the output. The loss function and optimiser were ours. But the weights — the actual knowledge, the thing that makes a prediction possible — those were discovered, not designed. The network found them by being wrong thousands of times and adjusting.

That’s the idea this whole series is built on. Not a model we wrote. A model that wrote itself, in a language of numbers, from nothing but examples and error.

In the next post, we’ll dive into why more parameters isn’t always best. And like with everything, bigger takes up more memory and more GPUs. So choose wisely.

My secondary school maths teacher, Miss Smith couldn’t be messed with. She wasn’t a big woman, and she didn’t raise her voice, but she had a way that the other teachers lacked — a quiet certainty that made you sit up straighter without quite knowing why. To me, maths was never really about numbers. It was puzzles. Logical puzzles, written in a language of numbers, functions and variables, waiting to be solved. Get the language right, and the answer had to follow. There was no arguing with it, no opinion to have. It simply was. It was then I formed the idea everything can be described in numbers. The “code” of the universe. The basis of AI.

That same love of numbers and logic — the idea that a well-posed problem contains its own answer, if you can only speak its language — is the thread that runs from those classroom puzzles all the way to the earliest machine learning algorithms, some of which trace their roots back centuries. Linear regression. Bayes’ theorem. Decision trees. K-Nearest Neighbour. And more besides. These aren’t relics. They sit quietly inside a huge proportion of the Python solutions doing real work in the world today, and they still deserve their place at the table. If you want to predict next quarter’s revenue from ten years of clean, structured data, you very rarely need anything more exotic than a well-fitted regression. Boring AI wins every time.

But somewhere in the twentieth century, ambition outgrew the puzzle-solver. Statistical models are, at their core, an exercise in description — here is the shape of the data, here is the line that best fits it, here is the probability given what we’ve already observed. What researchers wanted next was something different in kind, not just in degree: a machine that didn’t just describe patterns in data, but learned the way a brain does. Not a better equation. A different kind of thing entirely.

In this series I want to explore what I’m calling Neo-ML — neo machine learning — and how it has evolved to give us deep learning, computer vision, generative AI, and the ability to talk to our data and have it talk back in natural language. We’ll move from the artificial neuron through feedforward networks, into convolutional networks and computer vision, on to recurrent networks and long short-term memory, and finish at autoencoders and the doorway to generative AI. I’ve just coined a new term — Neo-ML — and by the end of this series you’ll understand exactly why it deserves one.

Let’s start this series with the round dot you see in most deep learning diagrams – the node, or neuron. And it’s just a mathematical equation in software. A function.

Part 1: The Artificial Neuron (Node)

The Biological Inspiration

The story starts, as so many things in computing do, with a question that sounds almost philosophical: can we build a machine that thinks?

In 1943, a neurologist named Warren McCulloch and a mathematician named Walter Pitts sat down and asked a more tractable version of that question. They weren’t trying to replicate consciousness. They were trying to model the simplest possible unit of biological computation — the neuron — in mathematical terms. Their paper, A Logical Calculus of the Ideas Immanent in Nervous Activity, described a neuron as something that receives signals, adds them up, and fires if the total crosses a threshold. Binary. On or off. Yes or no.

It was a brutal simplification of what a real neuron does. But it was the right simplification. It captured the essential logic: inputs come in, something happens in the middle, and a decision comes out.

That structure — inputs, transformation, output — is the skeleton of every artificial neuron ever built. Everything else in this series is an elaboration of it.

What an Artificial Neuron Actually Does

Let’s make it concrete.

Imagine you’re trying to decide whether to take an umbrella when you leave the house. You look at a few things: the sky looks grey, the forecast says 70% chance of rain, and you remember it rained yesterday. Each of those is an input. But they don’t all carry the same weight. The forecast matters more than the colour of the sky. Your memory of yesterday matters least of all.

An artificial neuron works the same way. It takes a set of inputs — numbers, each representing some feature of the world — and multiplies each one by a weight. The weight is the neuron’s way of saying how much it cares about that particular input. It then adds all those weighted inputs together, throws in a small constant called a bias, which lets it shift its behaviour even when all inputs are zero, and passes the result through a function that decides what to output.

In plain terms: the output equals an activation function applied to the weighted sum of the inputs, plus a bias. That last piece — the activation function — is where things get interesting.

The Activation Function: Where Non-Linearity Lives

If you removed the activation function and just kept the weighted sum, you’d have a linear model. Useful, but limited. The whole point of neural networks is that the real world is not linear. The relationship between the pixels in a photograph and the word “cat” is not a straight line. Neither is the relationship between a sentence and its sentiment, or between a patient’s symptoms and their diagnosis.

The activation function introduces non-linearity. It takes the raw weighted sum and squashes it, bends it, or clips it into a shape that lets the network learn curved, complex relationships.

The earliest activation function was the step function — exactly what McCulloch and Pitts described. If the sum crosses the threshold, output 1. Otherwise, output 0. Clean, but not differentiable, which matters enormously when you want to train the network (more on that in the next post).

The sigmoid function came next. It takes any input and maps it smoothly to a value between 0 and 1, producing an S-shaped curve. For a long time, it was the default choice. It has problems at the extremes — the gradients become vanishingly small, which makes learning slow — but it was a significant step forward.

Close behind sigmoid came the hyperbolic tangent, or tanh. It solves a similar problem in a similar shape — another smooth S-curve — but where sigmoid squashes everything into a range between 0 and 1, tanh maps to a range between -1 and 1. That zero-centring matters more than it sounds like it should: because tanh’s outputs can be negative, the weights in the next layer get pushed both up and down during training rather than only ever nudged in one direction. In practice, that tends to make networks converge faster than sigmoid does. It doesn’t solve the vanishing gradient problem — the curve still flattens out at the extremes, just like sigmoid’s does — but for a long stretch of the field’s history, tanh was the safer default of the two.

Today, the most widely used activation function is something almost embarrassingly simple: the Rectified Linear Unit, or ReLU. If the input is negative, output zero. If it’s positive, output the input unchanged. That’s it. No smooth curve, no elegant formula. Just a hinge. And yet ReLU, and its variants, power the majority of modern deep learning systems. Sometimes the right answer really is the simple one.

Why This Is a Different Kind of Thing

Here is the point that the name Neo-ML is trying to capture.

Classical machine learning algorithms — regression, decision trees, support vector machines — are built around explicit mathematical relationships. You, the engineer, decide what features matter. You decide how they relate to the output. The algorithm finds the best parameters to fit those relationships to the data. The structure of the model is yours. The algorithm fills in the numbers.

An artificial neuron — and the networks built from them — works differently. You don’t specify the relationships. You specify the architecture: how many neurons, how they’re connected, what activation functions they use. Then you show the network data, and it learns the relationships itself. The weights and biases are not chosen by you. They are discovered, through a process of trial and error at a scale no human could manage, by the training algorithm.

This is not a quantitative difference. It is a qualitative one. The model is not a description of the world that you wrote. It is a description of the world that the machine wrote, in a language of numbers that no human fully reads.

That shift — from models we design to models that design themselves — is what makes neural networks a different kind of thing. Not smarter than classical ML in every situation. Not always the right tool. But categorically different in what they are and how they work.

The Perceptron: The First Learnable Neuron

McCulloch and Pitts gave us the structure. But their neuron couldn’t learn. The weights were set by hand. It took another fifteen years, and a psychologist named Frank Rosenblatt, to build the first neuron that could update its own weights based on experience.

In 1958, Rosenblatt introduced the Perceptron. The idea was simple: show the neuron a labelled example, let it make a prediction, compare the prediction to the correct answer, and nudge the weights in the direction that would have made the prediction better. Do this thousands of times, across thousands of examples, and the weights gradually converge on something useful.

The Perceptron learning rule is the ancestor of every training algorithm in modern deep learning. Backpropagation, gradient descent, Adam optimisers — all of them are elaborations of the same core idea: measure the error, and adjust the weights to reduce it.

Rosenblatt was famously optimistic about what the Perceptron could do. Too optimistic, as it turned out. In 1969, Marvin Minsky and Seymour Papert published Perceptrons, a rigorous mathematical analysis that showed a single-layer Perceptron could not solve problems that weren’t linearly separable. The classic example is XOR — a logical operation so simple a child can understand it, but one that a single Perceptron cannot learn. The field went quiet for a decade. The first AI winter had arrived.

The solution, when it came, was obvious in retrospect: stack the neurons. Put one layer’s outputs into the next layer’s inputs. Build depth. That is the feedforward neural network, and it is where we go next.

What We’ve Established

A single artificial neuron is a small, precise thing. It takes inputs, weights them, sums them, applies a non-linear function, and produces an output. It can learn — in the sense that its weights can be adjusted to improve its predictions. And it is, in a meaningful sense, a different kind of computational object from the statistical models that came before it.

One neuron, on its own, is not very impressive. It can draw a line through data. It can make a binary decision. It is, in the end, not so different from logistic regression.

But connect enough of them together, in the right way, and something remarkable happens. The network stops drawing lines and starts drawing boundaries of arbitrary complexity. It stops recognising simple patterns and starts recognising faces, voices, sentences, intentions.

That emergence — the way complexity arises from simplicity, the way the whole becomes vastly more than the sum of its parts — is what the rest of this series is about.

Miss Smith would have approved. It turns out the universe is still a puzzle. We’ve just found a new way to speak its language.

Next up: Feed forward Networks