30 standalone questions on what training actually does — loss, gradient descent, the learning rate, backpropagation and overfitting.
Audience: absolute beginners. No maths beyond multiplying and subtracting; every number here can be checked on paper.
Source: the bottom half of the session-one whiteboard — predict, reconcile, recalibrate, loop — given the names the field uses for it.
How to use this: every question stands alone. Pick an option, then read the answer. The sections follow one training step from guess, to error, to blame, to nudge.
A · Nothing new is being introduced
Q1The board said predict, reconcile, recalibrate, loop. What does the field call those four things?
Answer: (2)Predict becomes the forward pass; reconcile becomes the loss; recalibrate splits into backpropagation (work out who is to blame) and gradient descent (nudge everything downhill); the loop becomes training. Nothing new is being introduced — you are getting names for steps you already understand.
Q2The board used the word experiment. Why is that the right word for what training does?
Answer: (2)Nobody designs the final weights. They are discovered, by experiment, at enormous speed. That is a genuinely useful thing for a non-technical audience to hold, because it explains why timelines are hard to predict.
B · Loss — one number for how wrong you are
Q3Someone tells you a model is "good on price accuracy, mediocre on new cars, excellent on hatchbacks, poor on diesels." Did the last change help or hurt?
Answer: (2)Which is exactly why loss must be one number. Boil it down to a single figure and the question becomes trivial: did the number go down? Every technique, every architecture and every GPU exists to make that one number smaller.
Q4Standard loss adds up the errors squared. Missing by 10 therefore counts as 100 rather than 10. Why?
Answer: (2)It is not fifty times worse, it is catastrophically worse — and squaring builds that judgement in. Preventing negatives is a side effect, not the reason.
Q5What is the consequence of squaring that is worth flagging?
Answer: (2)And that is a choice, not a law. If you would rather the model ignored the occasional wild outlier, you pick a different loss — which is a business decision wearing a technical hat.
C · Gradient descent — walking downhill in fog
Q6You are somewhere on a hillside in thick fog. You want the bottom of the valley and can see one step in any direction. What do you do?
Answer: (2)That is gradient descent. That is the whole algorithm. Everything else is detail. Where you stand = the current weights. Your height = the loss. The slope under your feet = the gradient. The size of your step = the learning rate.
Q7Why not simply try every combination of weights and pick the best?
Answer: (2)The fog is not a shortcut; it is the only option there has ever been. You will never see the landscape, so stepping downhill by feel is not a compromise — it is the method.
Q8Three data points where the real rule is y = 3x, and the machine must find the 3 starting from w = 1. After five steps w reaches 2.974 and the loss falls from 38.667 to 0.007. What has just happened?
Answer: (2)Not a metaphor for it — that is training. It found the answer without ever being told it, only whether it was getting warmer. Real systems do the identical thing across a billion weights at once, a few million times.
Q9A real system does not wiggle each weight and re-measure the loss. Does that make the wiggle explanation a simplification?
Answer: (2)With a billion weights, wiggling each one and re-measuring would take forever. The intuition is not a simplification; it is what the calculus computes.
D · The learning rate
Q10The step size has a name. What is the everyday analogy that captures it?
Answer: (2)Everyone has lived through this one. Get it right and you are comfortable in three adjustments; get it wrong in either direction and you either overshoot endlessly or never arrive.
Q11On the same three data points from the same start, a learning rate of 0.10 bounces across the valley but survives. A rate of 0.11 sends the loss climbing: 49 → 62 → 79 → 100 → 127. What does this show?
Answer: (2)Not slightly worse — broken, and getting worse with every step. It is climbing out of the valley rather than descending into it.
Q12What follows from that knife-edge for a business audience?
Answer: (2)Which is why AI projects have unpredictable timelines. Nobody can tell you the right learning rate in advance — it is found by experiment, which is the board's word again. And a very small rate is not a safe default: at 0.001, w moved from 1.000 to just 1.114 in three steps, with 3.0 still to reach.
E · Backpropagation — who is to blame?
Q13A customer sends a dish back. Nobody hands the chef a note saying "your sauce was 12% over-reduced." What is the only measurement available?
Answer: (2)So the blame has to be traced backwards: head chef gets most of it, sous-chef some, the supplier a little — each adjusting in proportion to how much they contributed to the final complaint. Nobody was told the right answer; each was told their share of the wrongness.
Q14Why is it called back-propagation?
Answer: (2)The hidden layers never see the right answer. Not once. They only ever receive their share of the blame from the layer in front of them.
Q15Two weights in a chain. Input 1 goes through w1 = 1, giving h = 1, then through w2 = 2, giving output 2 against a target of 6 — an error of −4. The blame on w2 is −4 and the blame on w1 is −8. Is w1 twice as wrong as w2?
Answer: (2)This is the one idea people most often get backwards about backpropagation. Nudging w1 moves h by x, and then that change is multiplied by w2 on its way out. Blame is apportioned by influence on the final output, not by how far off a weight happens to be.
Q16Why must the calculation run backwards rather than forwards?
Answer: (2)There is no way round it. The calculation starts at the output — the only place reality speaks — and works its way back to the front, layer by layer.
Q17A large deal closes after eight meetings, three emails and a dinner. Which one closed it? Why is this the same problem?
Answer: (2)Same problem, genuinely hard for the same reason. One measurement at the end, many contributors, and credit that has to be inferred rather than observed.
F · Why there is no formula
Q18A spreadsheet function fits a straight line to car prices instantly, exactly, first time — no fog, no steps, no learning rate. Why can a neural network not do that?
Answer: (2)Once you stack layers with a "throw away the negatives" step in between, the problem stops having a closed-form solution. There is nothing to solve. You can only walk downhill.
Q19Does a trained network give you the best possible network?
Answer: (2)Run the training again from a different random start and you get a different model, with slightly different answers, and no way to say which is truly better.
Q20Your team retrains last month's model on the same data and gets slightly different answers. Is something broken?
Answer: (2)Worth knowing before it turns into a panicked email six months from now. It is also why reproducibility in this field is genuinely hard, and why "we retrained and it behaves a bit differently" is an ordinary sentence rather than an alarm.
G · The two ways it goes wrong
Q21You walk downhill in fog and the ground stops falling away. Have you reached the bottom?
Answer: (2)The everyday version: settling into a job that is comfortable and fine, and never finding out whether the better one existed, because looking would mean climbing. What is actually done about it — several random starting points, a bit of momentum to roll through shallow dips — is not a guarantee, and the field has largely made peace with a good-enough valley.
Q22A student memorises ten years of past exam papers perfectly and scores 100% on all of them, then fails when the exam asks something slightly new. What is the machine learning name for this?
Answer: (2)Give a model enough weights and it will memorise your training data perfectly — loss of zero — and be useless on anything it has not seen. It never learned the subject; it learned those papers.
Q23A model scores 99% on the data it trained on and 61% on data it has never seen. What happened, and what should you do?
Answer: (2)The gap between the two figures is the diagnosis. More training would make it worse, not better — it would memorise harder.
Q24What is the one question to ask a vendor quoting "96% accuracy"?
Answer: (3)Hold some examples back, never train on them, then check the model on those. If a vendor quotes only one accuracy figure, they have not done this — and the number they are quoting is close to meaningless.
Q25A crude but useful instinct about model size. Which is it?
Answer: (2)Ten cars and three numbers to learn is healthy. Ten cars and ten numbers to learn would fit all ten cars perfectly and be worthless on the eleventh. Crude, but the right direction.
H · Reading the numbers
Q26A model's loss goes 40 → 22 → 12 → 7 → 4. Is training working?
Answer: (2)Diminishing improvements are exactly what approaching a minimum looks like. Near the bottom you want to creep rather than stride, which is why the steps get smaller as the slope flattens.
Q27A model's loss goes 40 → 62 → 91 → 140. What is the most likely single cause?
Answer: (3)Recall that 0.10 worked and 0.11 did not. A rising loss is the signature of overshooting, and it is one of the few symptoms in this field with a reliable first diagnosis.
Q28What does convergence mean?
Answer: (2)That last one describes an epoch — one complete pass through all your examples. Which surprises people: a model does not see your data once but hundreds of times. The data does not change; the weights do. It is revision for an exam — same syllabus, tenth pass, known better each time.
Q29What is a batch?
Answer: (2)Updating after every single example is noisy and slow; updating only after all of them is expensive. A batch is the practical middle.
Q30Put the loop in order.
Answer: (2)Guess with the weights you have. Get one number for how wrong that was. Share the blame backwards through the layers in proportion to influence. Nudge every weight a little way downhill. Repeat a few million times until the loss stops falling. The board had it right; the field just gave the steps longer names.