Four numbers disagree about the same ten laptops

MAE, MSE, RMSE and R squared are not four opinions. They are four questions asked about one set of gaps, and on this tiny table they answer 13, 244, 16 and 0.97, or 76, 18005, 134 and minus 1.33, depending on whether you include one laptop with a cracked screen.

10 rows · 5 columns · 6 to learn from · 4 held back

the whole dataset, and every gap the rule leavesclick any laptop
every number on this page is computed from these ten rows

Price this laptop, then find out what it sold for

Three years old. 8GB of memory. Battery at 80 percent of its original capacity. Somebody is about to buy it. What is it worth?

Commit to a number before you read on. Drag the marker, press the button, and the actual selling price appears next to it.

Whatever you said, the distance between your number and the real one is the only thing this entire page is about. It has a name: the error. Everything that follows, all four metrics, is arithmetic performed on a pile of those distances.

figure 1 · your guess against the truthdrag, then reveal
Drag, then reveal. The gap is drawn in rust. Nothing about it cares whether you guessed high or low, only how far off you were.
picture it

A tailor guessing your inside leg. She says 32, the tape says 31. The tailor is not wrong in a way that has a direction anybody cares about, she is out by one inch, and if she is out by one inch on every customer all day she has a very different problem from being out by nine inches on one of them.

Guessing one laptop is a party trick. The moment you have ten, you need a rule that produces a number for any laptop, including ones you have never seen.

A line is a rule you can check, and a feeling is not

Here are six laptops, plotted by age against price. There is an obvious slope in them: older is cheaper. Turn that into a rule by laying a straight line through the cloud.

Drag the slider. The line pivots. Under it, the total of every squared gap moves up and down, and your job is to make that total as small as it will go. When you are close, press the button and the arithmetic will do it exactly.

figure 2 · six points, one linesum of squared gaps: 0
Drag until the total stops falling. The line always pivots through the average laptop, so there is exactly one thing to tune and one number to watch.

The rule the arithmetic lands on, rounded so you can check it on paper:

price = 961112 × age
a three year old laptop → 961 − 112×3 = 625
a six year old laptop → 961 − 112×6 = 289
picture it

The line is not trying to touch the points. It is trying to be the least wrong about all of them at once, which usually means touching none of them. A rule that passed through every point would be memorising the six laptops rather than learning anything about laptops.

That rule now prices any laptop you hand it. Which raises the question the rest of this page exists to answer: how do you know it is any good?

Four of the ten rows were hidden from the rule on purpose

The table has ten laptops. The line you just fitted only ever saw six of them.

The other four were held back. The rule has no knowledge of them, was not adjusted to suit them, and cannot have memorised them. That makes them the only honest way to ask whether the rule works on a laptop it has never met, which is the only thing anybody actually wants to know.

Click any row to see where it sits.

figure 3 · ten rows, five columns, one splitclick a row
Click a row, or switch the view. Age drives the price here; memory and battery are along for the ride, which is deliberate and comes back at the end.
picture it

Handing a student the exam paper a week early and then being delighted by the marks. A score on questions you already showed them measures your generosity, not their understanding. The four held-back laptops are the questions they have not seen.

So there are two piles of gaps now: six from laptops the rule studied, four from laptops it has never met. Those gaps are about to be turned into numbers.

One gap, four things you can do to it

Take a single held-back laptop and watch what happens to its gap. Every metric on this page is one of these operations, repeated over four rows and divided by four.

figure 4 · what a gap is forlaptop H
Each step changes the drawing. Click a step to jump to it.

What actually happened

Laptop H is three years old and sold for €600. That is a fact about the world, and no model can argue with it.

What the rule claimed

961 − 112×3 = €625. The rule never saw this laptop. This is a genuine prediction.

The gap between them

600 − 625 = −25. Negative because the rule asked too much. The sign tells you the direction; most metrics throw it away immediately.

Strip the sign: the absolute error

|−25| = 25. Twenty five euros out. This is the number MAE averages, and it is measured in euros, which is why it is the easiest of the four to say out loud.

Or square it

(−25)² = 625. Drawn as an actual square, because that is what it is: a gap of 25 becomes an area of 625. Double the gap and the area quadruples. That single fact is the whole personality of MSE.

picture it

Two ways to fine a late train. Charge a euro a minute and a ten minute delay costs ten. Charge a euro per minute squared and the same delay costs a hundred. Neither is wrong. They are different opinions about whether one disaster is worse than several annoyances.

Four laptops, four gaps. Here they are, and they are the raw material for everything that follows.

laptop   actual   predicted    gap    |gap|    gap²
G        620         625     −5       5      25
H        600         625    −25     25     625
I        410         401      +9      9      81
J       470         737   −267   267  71289

Three of those gaps are small. One is not, and it is about to dominate everything.

MAE is the average gap, and it is in euros

Add the four absolute gaps. Divide by four. That is the entire calculation, and the answer comes out in the same unit as the thing you predicted, which means you can say it to a person who has never heard of a model.

MAE = 1/n × Σ |y − ŷ|

= ( 5 + 25 + 9 + 267 ) / 4
= 306 / 4
= 76.5   euros, on average, out

Click each bar below to add it to the running total, and watch the average move. Three of the four barely shift it. The fourth throws it across the figure.

figure 5 · adding up the absolute gaps0 of 4 added
Add them one at a time. After three laptops the average is 13. The fourth takes it to 76.5 on its own.
picture it

Your average commute this week: 30, 32, 29 and 95 minutes. The average is 46, a number that describes not one single day of your week. MAE is honest about the size of the typical miss and says nothing at all about whether the misses are alike.

MAE treats a gap of 267 as exactly 267 units of badness, no more. Sometimes that is right. Often it is not, because being wrong by 267 euros on one laptop is not the same kind of problem as being wrong by 5 on 53 of them.

MSE squares every gap, so the worst row shouts loudest

Same four gaps. Instead of stripping the sign, square them, which strips the sign anyway and does something else on the way: it makes big gaps enormously more expensive than small ones.

MSE = 1/n × Σ (y − ŷ

= ( 25 + 625 + 81 + 71289 ) / 4
= 72020 / 4
= 18005   euros squared, which is not a thing anybody can picture

RMSE = √MSE = √18005 = 134.2   euros again

Look at where that 72020 comes from. Laptop J contributes 71289 of it. The other three laptops together contribute 731, which is one percent of the total. In MAE, J was 87 percent of the sum. Squaring took it to 99 percent.

figure 6 · the gaps, drawn as areassquares to scale
Switch between lengths and areas. As lengths J is ten times the others. As areas it is over a hundred times, and the small three almost vanish.
picture it

Insurance. Ten scratched bumpers cost the insurer far less than one written off car, so the premium is not set by the average dent. Squaring is how you say out loud that one catastrophe matters more than a hundred inconveniences, and RMSE then converts the answer back into money so you can quote it.

RMSE is the square root of MSE, which puts it back into euros and makes it sayable. It is not the same as MAE, and the difference between them is a diagnostic in its own right: RMSE 134 against MAE 76 is a ratio of 1.75, and a ratio that high is the signature of one row doing most of the damage. When the ratio sits near 1.1, your errors are all much of a muchness.

MAE asks how far off you usually are. MSE asks how bad your worst days get.

Both of those are absolute numbers. Neither of them tells you whether 76 euros is good, and that needs something to compare against.

R squared scores you against the laziest model there is

Is 76 euros of average error good? The question has no answer on its own. It depends entirely on what the alternative was.

So R squared invents an alternative and beats you with it. The alternative is the laziest model imaginable: ignore age, ignore memory, ignore everything, and always predict the average price. That model has no idea what a laptop is. Whatever you have built should beat it.

R² = 1 − SSres / SStot

SS res  = your squared gaps   = 25 + 625 + 81 + 71289 = 72020
SS tot  = the lazy model's squared gaps, against the mean of 525
          = 95² + 75² + 115² + 55² = 30900

R² = 1 − 72020 / 30900 = 1 − 2.331 = −1.33

Negative. That is not a broken calculation, it is the calculation working. R² of 0 means you exactly matched the lazy model. Above 0 means you beat it. Below 0 means you lost to a model that does not look at the data at all.

figure 7 · your gaps against the lazy model's gapsboth sets shown
Compare the two sets of sticks. Three of yours are shorter than the lazy model's. The fourth is so much longer that it outweighs all three.
picture it

A weather forecaster who says "same as yesterday" every single day is the baseline. If your satellites and supercomputers cannot beat that, the equipment is not the point. R² is the ratio of your mistakes to that person's mistakes, flipped so that bigger is better.

On the six training laptops this same model scores 0.99. On the four it had never seen, it scores minus 1.33. Same model, same arithmetic, two wildly different verdicts, and only one of them was ever going to reach a customer.

One row out of ten decides whether this model looks excellent or worthless

Here are all four metrics, live, over the four held-back laptops. The switch drops laptop J out of the test set and puts it back.

Nothing else changes. Same rule, same six training rows, same three other laptops. Watch which metrics shrug and which ones fall over.

figure 8 · the same model, judged twice4 test laptops
Flip the switch. MAE moves by a factor of 6. RMSE moves by 9. R² travels from 0.97 to below zero. Every one of those numbers is arithmetically correct.
with J     MAE 76.5   MSE 18005   RMSE 134.2   R² −1.33
without J  MAE 13.0   MSE 243.7   RMSE 15.6   R² 0.97
on the six training rows  MAE 20.7   MSE 538.3   RMSE 23.2   R² 0.99
picture it

A restaurant with four reviews: three at five stars and one at one star from a diner who got food poisoning. The average rating is 4.0, which describes nobody's evening. Whether you quote the average or the worst case is not a statistical question, it is a question about what you are responsible for.

The temptation here is to call J an outlier, delete it, and report the good numbers. That is the single most common piece of quiet dishonesty in applied machine learning, and it is worth being precise about why it is dishonest.

You do not get to remove a row because it makes your model look bad. You get to remove it if you can say what made it different.

Laptop J is not noise, it is a column you never collected

Look at J in the table. Two years old, 16GB, battery at 86 percent. By every column available it should have sold for around 737 euros, and it went for 470.

The reason is not in the table. It had a cracked screen. There is no column for that, so the model could not know, and no amount of clever modelling on these five columns will ever recover it. The error is not the model being stupid. The error is the dataset being incomplete, and the metric is the only thing telling you so.

Add the column and the problem dissolves. That is what the button below does.

figure 9 · the column that was missing5 columns
Add the sixth column. J stops being mysterious and becomes ordinary: cracked screens sell at a discount, and now the rule can say so.
picture it

A doctor whose notes have no field for "patient is a smoker" will keep being surprised by the same patients. The surprise is information: it is telling you which question your form forgot to ask. A big residual is a lead, not a nuisance.

So the working habit is not "watch one metric". It is: split before you fit, quote MAE and RMSE together because the gap between them tells you whether one row is carrying the damage, check R squared against zero rather than against 1, and go and look at your largest residual by name.

On ten rows you can do all of that by eye. On ten million the arithmetic is identical and the largest residual still has a name, which is the entire reason to learn it here.

What the four numbers were actually for

MAE told you the typical miss in euros. MSE told you the worst miss dominated. RMSE said the same thing in money. R squared said you had lost to a model that ignores the data. None of them told you about the cracked screen, and all four of them pointed straight at it.