In 2012, a team at the University of Toronto entered an image-recognition contest called ImageNet and roughly halved the error rate everyone else had been stuck at. Their system looked at photographs and guessed what was in them. The approach itself was not new. It was a neural network, an idea that had been kicking around since the 1950s and dismissed more than once. What changed was scale: a much larger dataset, gaming GPUs, and a deeper stack of layers.
That gap between old idea and sudden breakthrough is the most useful thing to grasp about neural networks. They are not magic and not a single algorithm. They are a way of building a function out of very simple parts, then letting data turn the dials.
The basic unit does almost nothing
Take one artificial neuron. It receives a handful of numbers, multiplies each by a weight, sums the results, adds a bias, and passes the total through a small squashing function. That’s it. No reasoning, no symbol manipulation.
The inputs might be pixel brightness values, word frequencies, or the outputs of other neurons. The weights are the part that matters. A weight of 0.9 means this input carries a lot of signal. A weight near zero means ignore this one.
Alone, a neuron can only draw a straight boundary through data, which is enough to sort email into spam and not-spam if the two groups happen to be neatly separable. Real problems rarely are. So neurons get stacked.
Why layers change the maths
Put neurons in layers and each layer’s output feeds the next. In an image classifier, early layers tend to respond to edges and colour gradients. Middle layers combine those into corners, curves, and textures. Later layers assemble them into object parts. By the final layer the network is effectively voting on ears, wheels, and window frames.
No one writes those intermediate features by hand. They appear because the training process needs weights that make the final answer correct, and building useful intermediate representations is the cheapest route there. If you want the fuller picture of what’s really happening inside a deep learning model, that’s worth reading alongside this.
How a network actually learns
Show the network a photo of a dog. It outputs a probability distribution: 0.72 cat, 0.28 dog. The truth is dog, so a loss function scores how wrong that is. Then comes the clever part.
Backpropagation runs the chain rule of calculus backwards through every layer to work out how much each individual weight contributed to the error, and in which direction. Gradient descent then nudges every weight a fraction in the direction that reduces the loss. That fraction is the learning rate. Too small and training crawls; too large and it bounces around and never settles.
Repeat that loop a few million times over a few million examples and the weights drift into a configuration that generalises. A modern training run can involve billions of weight updates. The maths behind each one is elementary. The scale is not.
A common misconception is that the network stores the training data and looks things up. It doesn’t. It compresses patterns into a few hundred million or billion numbers, which is why it can handle a photo it has never seen, and also why it can fail strangely on one that is slightly unusual.
Depth, scale, and a surprise nobody planned for
Adding layers helped, then helped a lot. Somewhere past a few dozen layers, models started doing things their designers hadn’t explicitly trained them for: translating between languages they’d only seen incidentally, doing arithmetic from text, following multi-step instructions. The general shape of the trend became known as scaling laws, and the race to build bigger models turned into one of the strangest stories in the field, captured well in this account of Ilya Sutskever and the models he built before he left.
Scale is not unlimited, though. Training runs cost tens of millions of dollars, and returns per dollar are flattening. Much of the current work is about doing more with less: better data curation, smarter architectures, and fine-tuning small models for narrow jobs.
Where neural networks fall apart
The failure modes are consistent enough to list.
- Overfitting. Give a network too few examples and it memorises them, scoring 99% on training data and 61% on anything new. Regularisation and more data are the usual fixes.
- Distribution shift. A model trained on bright, professionally lit product shots degrades quickly on dim phone snapshots. Live deployments drift as the world changes.
- Spurious shortcuts. A well-worn cold-war anecdote involves a tank detector that had learned to recognise cloudless skies rather than tanks, because every positive example had been photographed on a sunny day.
- Label noise. Human annotators disagree with each other roughly 10% of the time on many tasks. The network learns the disagreement as if it were signal.
- Adversarial sensitivity. Small, deliberate pixel changes that look like nothing to you can flip a confident prediction completely.
None of these are bugs you can patch out. They are consequences of learning from examples rather than rules, and they are why the honest limits of machine learning deserve more airtime than the demos.
The unglamorous part of the job
Practitioners spend most of their time on data, not architecture. Deduplicating, fixing mislabelled rows, checking whether the training set accidentally includes the test set, deciding what correct even means for an ambiguous case. A reasonable split is 70% data work, 20% evaluation, 10% modelling.
Evaluation is the part that quietly decides whether a project ships. Aggregate accuracy hides a lot. A fraud model at 99% accuracy sounds fine until you notice it misses half the fraud and flags thousands of legitimate transactions. Per-segment breakdowns, confusion matrices, and a held-out test set that mirrors real deployment conditions matter far more than one headline number. That pipeline, rather than the model, is usually the bottleneck, and this walkthrough of how AI systems actually work end to end covers it well.
You use them dozens of times a day
Speech recognition when you dictate a message. The ranking layer behind a search box. Face unlock. Photo search that finds every picture of your dog. Machine translation, spam filtering, fraud scoring, and the noise reduction on a call. Medical imaging tools that flag potential tumours for a radiologist to review.
Most of these are small, narrow networks doing one job well rather than anything resembling general intelligence. That distinction matters when you read headlines, and a plain-English guide to artificial intelligence is a reasonable place to keep your bearings.
What separates a working system from a demo
Three things, in practice. A clear definition of the decision being made and who is accountable for it. A monitoring setup that catches drift before users do. And a fallback path, whether that’s a human reviewer, a rule-based check, or a graceful refusal when confidence is low.
Tuning a neural network to 90% accuracy is a weekend. Getting the last few points, and keeping them once the data shifts, is where the real work lives. Build the evaluation harness before the model. Collect examples of the failures you care about most, including the ugly ones nobody wants in the slide deck. Decide in advance what happens when the system is wrong, because it will be, and the people affected will notice long before your dashboard does.