How wrong?: measuring error with a loss

You set knobs by eye in chapter 1. A machine has no eye. It needs one number that says how wrong the rule is. This chapter builds that number.

By the end of this chapter, you will be able to

Say what a miss is, and point to one on a graph.
Turn five misses into one number that says how wrong a rule is.
Explain why one big miss counts for more than a few small ones.
Find the best knob setting by finding the bottom of a valley.

Real measurements wobble

In chapter 1 every dot sat right on the line. Real life is not that neat. This time someone clocked the same rider by hand, with a stopwatch. Each reading is a little off.

Try itTry to get the line through all five dots.

speed0.27 km per minute

distance = 0.27 × minutes

0 of 5 dots are on the line. You cannot get them all, because the dots do not sit in a straight row. Right now the biggest miss is at minute 10: the line is 1.64 km too low.

No setting is perfect any more. So the question changes. It is no longer “which setting is right?” It is “which setting is the least wrong?”

The red sticks are a start. Each one shows how far the line is from a dot, straight up or down. We will call that a miss.

A miss is the distance between what the rule says and what was measured.

Five misses, one number

Five misses are still five numbers. When you turn the knob, some get smaller and some get bigger. Is that better or worse? To decide, we need to roll them into one number.

We could just add up the sticks. But think about two lines. The first misses all five dots by 1 km. The second hits four dots, and misses the last one by 5 km. Both add up to 5. Yet the second line is badly wrong about that last dot. It should score worse.

Here is a trick that does this. Turn each miss into a square, with sides as long as the miss. Then look at the size of the square, not the length of the stick.

Try itEach miss is now a square. Turn the knob to make the wrongness as small as you can.

wrongness1.050

speed0.27 km per minute

squares: 0.36 + 0.09 + 1.21 + 0.90 + 2.69, shared between 5 dots = 1.050

Watch the squares. A miss twice as long makes a square four times as big.

A miss of 1 makes a square of size 1 × 1 = 1. A miss of 5 makes a square of size 5 × 5 = 25. So the first line scores 1 + 1 + 1 + 1 + 1 = 5, and the second scores 25. Big misses now count for much more than small ones.

Add up the squares, and share the total between the five dots. That gives one number. We will call it the wrongness. Small is good. Zero would be perfect.

In symbols

We now have two distances for every dot: the one the rule says, and the one that was measured. They need different letters. The measured one keeps the plain $y$ . The rule’s guess gets a little hat: $y^$ .

y^= w x

The rule from chapter 1, with a hat on its answer to show it is a guess.

miss = y^- y

The miss is the guess minus the measured value.

L = \frac{1}{n} i = 1 \sum n (y^_{i} - y_{i})^{2}

Read it from the inside out. Take the miss for dot number $i$ . Square it. The big $\sum$ means “add these up for every dot, from the first to the last”. Then divide by $n$ , the number of dots, to share it out.

$L$: the wrongness. The letter is short for “loss”.
$n$: how many dots there are. Here, 5.
$i$: which dot we are on: 1, 2, 3, 4 or 5.

It looks like a lot. But it is only what you just did with the squares: miss, square, add, share.

The valley

You found the lowest wrongness by trying settings and watching the number. Now let us see all the settings at once. For each place the knob can be, we will mark how wrong the rule is there.

Try itSweep the knob slowly from one end to the other. Each setting leaves a mark below, at its wrongness.

speed0.27 km per minute

You have tried 0 of 45 settings. Keep sweeping until the marks make a shape.

The marks make a valley. On the left, the line is too flat, and the wrongness is high. On the right, the line is too steep, and it is high again. In between there is one lowest point.

Learning means finding the bottom of this valley. The best knob setting is the one where the wrongness is lowest.

Look at what just happened. “Find a good rule” was a fuzzy job. Now it is a clear one: go downhill until you cannot go lower.

The catch

You found the bottom because you could see the whole valley. To draw it, you tried every setting of one knob. That was 45 tries.

With two knobs, you would need to try every pair: 45 × 45, about 2,000 tries. With twelve knobs it is more tries than there are grains of sand on Earth. A machine with billions of knobs can never draw its valley.

So the machine has to find the bottom without seeing the valley. It is like walking down a hill in thick fog. You cannot see the bottom. But you can feel which way the ground slopes under your feet. That is the next chapter.

You can now

Say what a miss is, and point to one on a graph.
Turn five misses into one number that says how wrong a rule is.
Explain why one big miss counts for more than a few small ones.
Find the best knob setting by finding the bottom of a valley.