Bends: neural networks and backpropagation

A weighted sum can tilt, but it cannot bend. This chapter adds one small part to the rule that lets it bend. Stack up enough of those parts, and the rule can take any shape at all.

By the end of this chapter, you will be able to

Say why a weighted sum, by itself, can never bend.
Build a bend from one part that stays silent and then wakes up.
Explain how many small bends add up to any shape.
Point at a neuron and say the two things it does: a weighted sum, then a cut-off.
Say what a neural network is.
Explain how a network finds which knobs to blame for a miss, by working backwards.

The tired rider

Chapter 4 ended with a problem. The rule charged 6 minutes for every hill. But a real rider gets tired. The first few hills are fine. After that, each one hurts more.

Here are nine real rides. They are all 20 km long, so only the hills differ.

Try itNine rides, each 20 km, with 0 to 8 hills. Try to get the straight line through all nine dots.

per hill6.0 min per hillbase50 min

time = 6.0 × hills + 50

This is chapter 4’s rule: 50 minutes, plus 6 for each hill. It is right for rides with up to 3 hills. After that the real rides take longer and longer.

Whatever you do with the two knobs, you get a straight line. One knob tilts it. The other lifts it. Neither can put a corner in it. A weighted sum is always straight.

More knobs of the same kind will not help. Add a hundred more things to look at, and the rule still charges one fixed price for each. To bend, the rule needs a new kind of part.

A part that wakes up late

Look at the dots again. Up to 3 hills, the old rule is right. The trouble starts after that. So we do not need to throw the old rule away. We need something that adds to it, but only on rides with more than 3 hills.

Here is a part that does that. It stays silent up to a point. Past that point it wakes up, and starts to add.

Try itThis is one new part, by itself. Move its two knobs and see what shape it makes. Then slide the ride along to see what it adds.

wakes up at4.0 hillssteepness8.0 min per hillthis ride6 hills

adds 8.0 × (hills − 4.0), but never less than 0

A ride with 6 hills is 2.0 past the wake-up point. The part is awake and adds 8.0 × 2.0 = 16 minutes.

The part has two knobs. One sets where it wakes up. The other sets how steeply it rises once it is awake. Before the wake-up point it adds nothing at all.

Silent, then awake. That corner, where the flat part meets the rising part, is the bend we were missing.

Straight plus late makes a bend

Now put the two together. Keep chapter 4’s straight rule, and add the new part on top.

Try itThe dashed line is chapter 4’s straight rule. The new part is added on top of it. Set the part’s two knobs so the solid line goes through every dot.

wakes up at6.0 hillssteepness2.0 min per hill

time = 6 × hills + 50 + 2.0 × (hills − 6.0, never less than 0)

4 of 9 dots are on the line. First put the wake-up point where the dots start to leave the dashed line. Then set how steeply it rises.

The straight rule did not change. It still charges 6 minutes a hill. The new part sits on top and stays out of the way until it is needed. Add a straight line and a late-waking part, and you get a line with a bend in it.

And the bend is not fixed in place. Its two knobs say where it is and how sharp it is. Knobs can be learned. So the machine can learn where the rider gets tired, just as it learned what a hill costs.

Many small bends make any shape

One part gives one bend. What if the shape we need is not a single corner, but a real curve? Use more parts. Each one wakes up at a different place and adds its own bend.

Here is a harder shape: the height of a road that goes over two hills. And here is a new trick. A part’s steepness knob can be negative. Then, when it wakes up, it pulls the line down. That is how the line gets back down the far side of a hill.

Try itThe dark dots are the height of a real road with two hills. The orange line is built from bending parts. Give it more parts.

how many parts

One part can only make a straight line here. It is off by about 36 m on a typical dot.

No single part is clever. Each one is a flat line with one corner. But enough simple bends, added together, can copy any shape: a hill, a wave, anything you can draw without lifting your pen.

This is the reason these machines are so widely used. You do not have to know the shape in advance. Give the machine enough parts, and it bends itself to fit.

Look inside

So far we have drawn what the machine makes. Now look at the machine itself. Here it is with six parts, drawn as six circles.

Try itSlide along the ride. Below the road is the machine that draws the orange line. Watch its six parts wake up one by one.

where on the ride9.0 km

At km 9.0, 3 of 6 parts are awake. Their shares add up to a height of 65 m. The real road is at 77 m.

Follow one number through it, from left to right.

The position goes in. Every part looks at it and asks: am I past my wake-up point? If not, it stays dark and sends on a zero. If so, it lights up and sends on how far past it is.

Then comes a move you know. Each part’s number is multiplied by a knob, and the results are added up. That is the weighted sum from chapter 4. The thick lines are big knobs. The dashed ones are negative.

Each part does two things: a weighted sum, then a cut-off at zero. Then a weighted sum joins the parts together. That is the whole machine.

How it learns: who is to blame?

In that last figure the knobs were already set. Who set them? Not a person. The machine did, with the loop from chapter 3: measure the wrongness, feel the slope for every knob, turn each knob a little downhill, repeat.

But there is a real puzzle here. In chapter 4, every knob sat right next to the answer. Turn the hills knob, and the guess moved straight away. It was easy to see what each knob did.

A network is deeper than that. Some knobs are tucked away inside a neuron. A neuron’s wake-up point does not touch the answer directly. It changes what the neuron sends. That is then multiplied by another knob. Only then does it reach the answer. So when the guess is wrong, which knobs are to blame, and by how much?

The answer is to start at the miss and work backwards. Take one spot on the road, and follow it through the three stages.

Try itOne spot on the road, one guess. Step through the three stages to see how the machine finds out which knobs to turn, and which way.

where on the ride9.0 km

The numbers run from left to right. The guess is 65 m. The real road is at 77 m. So the guess is 12 m too low.

Two simple rules did all of that.

Blame follows the wires. A knob on a wire that carried a big number gets a big share of the blame. It did a lot to make the answer what it was.

Silent neurons are let off. A neuron that sent zero had no part in this guess. Nothing you do to its knobs would have changed it. So it gets no blame, and it passes none back.

And “blame” is nothing new. It is the slope from chapter 3. A knob’s blame says which way to turn it, and how hard, to shrink the miss. We have just found a way to get it for knobs deep inside the machine.

Guess forwards. Pass the blame backwards. Turn every knob a little. Working out the blame this way, from the answer back towards the start, is called backpropagation.

Why backwards? Because each stage reuses the one before it. The miss is worked out once, and handed to every knob on the right. Each neuron’s blame is worked out once, and handed to every knob inside it. In a machine with a hundred layers and a billion knobs, that saving is what makes learning possible at all.

In symbols

The blame is a slope, so it is written like the slopes of chapter 3. For a knob on the way out, $v_{j}$ :

\frac{\partial L}{\partial v _{j}} = the miss, doubled 2 (y^- y) \times what neuron j sent h_{j}

Stage 2 of the figure. The 2 comes from the square in the wrongness. The figure left it out, because it is the same for every knob.

For a knob inside neuron $j$ , there is one more link in the chain:

\frac{\partial L}{\partial w _{j}} = the miss, doubled 2 (y^- y) \times its outgoing knob v_{j} \times 1 or 0 [awake?] \times what went in x

Stage 3. Each link is one hop back along the wires. If the neuron is silent, the middle link is 0, and the whole thing is 0.

Multiplying the links like this is called the chain rule. Backpropagation is the chain rule, done in an order that never works out the same link twice.

Watch it learn a shape

That was one spot on the road. To learn the whole road, the machine does it for all 41 dots, and adds up each knob’s blame. Then it takes one small step: every knob moves a little, each in its own downhill direction. That is one step. Then it does it again, thousands of times.

Here is a fresh network doing exactly that. Nothing is worked out ahead of time. Each step you see is the step from chapter 3.

Try itThis network has twelve neurons and starts out knowing nothing. Press the button. Every step, it guesses all 41 dots, passes the blame back, and turns every knob a little.

typical miss42 m

step 0 of 6000

Before learning, the line is almost flat. A typical miss is 42 m.

Think back to the fog. With one knob, the valley was a curve, and one slope told you which way to step. This network has 37 knobs. Its valley has 37 directions, and nobody can draw it. It does not matter. Backpropagation gives the slope in every direction at once, and the machine steps downhill in all of them together.

So “learning any shape” is not a new kind of learning. It is the ghost rider’s knob, 37 times over, with a clever way of feeling the slope.

What is still missing

We now have a machine that can learn any shape from examples. That is a big step. But look at what we have been feeding it: minutes, kilometres, hills. Numbers.

The machines this article is heading for work with words. And you cannot multiply a word by a knob. What is 0.5 times “bicycle”?

Before a network can read, words have to become numbers. And not just any numbers. That is the next chapter.

You can now

Say why a weighted sum, by itself, can never bend.
Build a bend from one part that stays silent and then wakes up.
Explain how many small bends add up to any shape.
Point at a neuron and say the two things it does: a weighted sum, then a cut-off.
Say what a neural network is.
Explain how a network finds which knobs to blame for a miss, by working backwards.