A model with a billion things to adjust is standing on a hillside in thick fog. It cannot see where the bottom is. It can only feel which way the ground tilts under its feet, take a step that way, and feel again. That is the whole of how models learn — and the rest of this page is that sentence, slowly.
Nothing new is being introduced here. If you have ever tasted a dish, decided it needed more salt, added some, and tasted again, you have run the entire algorithm. What follows is the field’s names for steps you already understand, plus the honest detail of how a machine does the “adjust it a bit” step when there are a billion things it could adjust.
Tap any step to see its three names.
One word deserves keeping: experiment. Every single step of training is a tiny experiment — change something slightly, measure whether it helped, keep the change if it did. Nobody designs the final numbers inside a model. They are discovered, by experiment, at enormous speed.
The loss is the machine’s report card, and it is a single number. Ramesh at the tea stall predicted 120 cups and sold 145 — a gap of 25. Next time, a gap of 7. That shrinking number is the entire point of training.
Why must it be one number? Here is a model scored four different ways, before and after a change. Decide whether the change helped.
With four measures pointing in different directions there is no way to say whether a change was an improvement. Boil it down to one number and the question becomes trivial: did the number go down?
The standard loss adds up the errors squared, which sounds like a technicality and is not. Drag the size of the miss.
This matches how being wrong actually feels. Pricing a car a little off is a rounding error. Pricing it wildly off loses you the customer and possibly the deal. It is not fifty times worse; it is catastrophically worse. Squaring builds that judgement in.
There is a consequence worth flagging: squaring makes the model obsessive about its worst mistakes, sometimes at the cost of the ordinary ones. That is a choice, and other choices exist. If you would rather the model ignored the occasional wild outlier, you pick a different loss — which is a business decision wearing a technical hat.
Here is the situation the machine is actually in. You are somewhere on a hillside. It is thick fog. You want to reach the bottom of the valley. You cannot see more than one step in any direction. What do you do?
Almost everyone says the same thing: feel which way the ground slopes under your feet, and take a step that way. Then feel again. That is gradient descent. That is the whole algorithm. Everything else is detail.
Tap each thing in the fog to see what it is in the machine.
Could you not just try every combination and pick the best? Set the size of the network and see.
Stepping downhill in fog is not a shortcut. It is the only option there has ever been.
Three data points. The real rule is that y is always three times x — but the machine does not know that. It has to find the three. Its rule is y = w × x, so it has exactly one thing to learn: w.
| x | y |
|---|---|
| 2 | 6 |
| 3 | 9 |
| 4 | 12 |
First, feel the slope. Wiggle w a little in each direction and see which way the loss falls.
Making w bigger makes the loss smaller, so downhill is up. One honest note: a real system does not wiggle each weight and re-measure. With a billion weights that would take forever. It uses calculus to get the slope of all of them at once, instantly. But the meaning is exactly the wiggle above. The intuition is not a simplification; it is what the calculus computes.
Now walk down. Take a step proportional to how steep it is — steep slope, big step; nearly flat, small step, because near the bottom you want to creep, not stride.
| step | w | loss |
|---|---|---|
| start | 1.000 | 38.667 |
| 1 | 2.160 | 6.821 |
| 2 | 2.647 | 1.203 |
| 3 | 2.852 | 0.212 |
| 4 | 2.938 | 0.037 |
| 5 | 2.974 | 0.007 |
Five steps. It has essentially found w = 3 without ever being told the answer — only whether it was getting warmer.
The step size has a name — the learning rate — and it is the most temperamental setting in machine learning. Everyone has lived the analogy: adjusting a shower tap. Turn it too much and you go from freezing to scalding and back, overshooting every time. Turn it too little and you stand there shivering for five minutes.
Below is the same valley and the same starting point, with the fog lifted so you can see what the machine cannot. Drag the step size.
The dotted path is where the walker actually goes. Everything to do with training that people find mysterious — why runs fail, why timelines slip — is largely this one dial being slightly wrong.
Push the dial past about 0.10 and watch. At 0.10 the walk bounces from one side of the valley to the other but still creeps inward. At 0.11 it bounces outward — the loss actively climbs, and the model gets steadily worse the longer you train it.
Two things follow. Training runs fail for reasons that have nothing to do with your data or your idea — a setting nobody outside the team has heard of was slightly too large. And this is a real part of the cost: “we are tuning hyperparameters” means “we are hunting for numbers like this one”, which is why this kind of project has unpredictable timelines.
Everything so far had one weight. Real networks have layers of them, and that creates a genuinely hard problem. A customer sends a dish back. It is bad. Who should change what?
The manager can observe exactly one thing: the customer is unhappy. Nobody handed the chef a note saying “your sauce was 12% over-reduced”. So the blame has to be traced backwards. Step it back through the kitchen.
Each person adjusts their own work in proportion to how much they contributed to the final complaint. Nobody was told the right answer. They were each told their share of the wrongness. That is backpropagation.
The hidden middle layers never see the right answer. Not once. They only ever receive their share of the blame from the layer in front of them.
Two weights in a chain. The input goes through the first, then the second, then out. The target is 6 and the output is 2, so the error is −4. Now share out the blame — and notice that the second weight’s value changes the first weight’s share.
The first weight carries more blame than the second — not because it is more wrong, but because whatever it does gets multiplied by the layer in front of it. Influence, not wrongness.
Look again at how the first weight’s blame was worked out: it needed the second weight’s value. You cannot work out the first layer’s blame until you already know the last layer’s. So the calculation starts at the output — the only place reality speaks — and works its way back to the front, layer by layer.
There is a version of this everyone recognises. A large deal closes after eight meetings, three emails and a dinner. Which one closed it? You only observed the outcome. To apportion credit you have to work backwards through the chain, giving each step a share based on how much it moved things along. Same problem, genuinely hard for the same reason.
Fair question, and there is a sharp answer. For a straight line, a formula exists. Mathematicians solved that in the 1800s: feed in the data, out comes the exact best answer. No fog, no steps, no learning rate.
For a neural network, no such formula exists and none ever will. Once you stack layers with a “throw away the negatives” step in between, the problem stops having a solution you can write down. There is nothing to solve. You can only walk downhill — and where you start decides where you finish. Train it again and see.
| A straight line | A network | |
|---|---|---|
| Is there a formula? | Yes | No |
| How you get the answer | Solve it, once | Step downhill, millions of times |
| How long | Instant | Hours to months |
| Do you get the best answer? | Yes, provably | No — you get a good one |
A trained network is not the best possible network. It is wherever the walk happened to stop.
You walk downhill in fog and eventually the ground stops falling away. You have reached a bottom — but is it the bottom? In fog you cannot tell whether the real valley is just over the next ridge.
Here is a landscape with two valleys, one genuinely deeper than the other. Drop the walker in different places and see where it ends up. It always goes downhill correctly. That is the problem.
The everyday version: settling into a job that is comfortable and fine, and never finding out whether the better one existed, because looking would mean climbing.
What is actually done about it: start from several different random places and see whether they agree, and give the walk a bit of momentum so it can roll through a shallow dip rather than stopping in it. Neither is a guarantee. In practice the field has largely made peace with this — a good-enough valley is usually good enough.
This is the one that costs real money, so it deserves time. Picture the student who mugs up past papers. Ten years of question papers, memorised perfectly. Full marks on every one. Then the exam asks something slightly new and they are lost — because they never learned the subject, they learned those papers.
Below are ten data points that genuinely came from a straight line, with a bit of noise. Give the model more and more numbers to learn, and watch both scores. These are real fits, computed on real numbers.
At ten numbers for ten points it fits the training data perfectly — an error of zero — and its error on data it has never seen is catastrophic. Fewer things to learn than examples to learn from is a crude rule, but it is the right instinct.
Notice too that the held-back error does not climb smoothly. It wobbles, dips, and only then explodes. Real curves look like this, which is exactly why you check rather than assume.
Ten questions. Everything you need is above, and the last two are the ones that would otherwise turn into a worried email six months from now.
Everything on this page is one cycle, running until the number stops falling. Step it round once.
That is it. Predict, reconcile, recalibrate, loop — the field just gave the steps longer names.
Two honest limits to carry away. Training does not find the best answer, only a good one it happened to walk into, and a different run finds a different good one. And a low loss is not the same as a useful model — a model that has memorised its training data has a loss of nearly zero and is worthless, which is exactly why the held-back test matters more than the headline number.