Agreement

A word is now a list of numbers. This chapter builds two small tools for working with such lists. The first gives one score for how well two lists agree. The second turns a set of scores into shares of 100%. Attention is made of exactly these two tools.

A score for a good match

In chapter 6, a machine had to say whether a word suits a gap. It did that with a weighted sum: the gap’s knobs times the word’s numbers, added up. We used it and moved on. Now we look at it properly, because it is the most-used move in the whole machine.

Start with something from everyday life. You want to pick a route for a ride. You know what you like. Each route has what it has. How well does a route suit you?

Write what you want as a list of numbers: a big number for things you love, a negative number for things you hate, zero for things you do not care about. Write what the route has as a second list, in the same order. Then go down the two lists together.

Look at what each pair does to the score.

You want it, and the route has it. A positive times a positive. The score goes up.

You hate it, and the route has it. A negative times a positive. The score goes down.

You do not care, or the route has none. Anything times zero is zero. That pair says nothing.

Multiply the two lists in pairs, and add up the results. The answer is one number that says how well the lists agree.

This is chapter 4’s weighted sum, and nothing more. What is new is how we look at it. In chapter 4, one list was “knobs” and the other was “the ride”. Here the two lists are equals. Neither one is the rule. They are two lists, and the sum asks how well they match.

The same score, seen as arrows

Chapter 6 turned a list of two numbers into a place on a map. Draw an arrow from the middle of the map to that place, and the list becomes an arrow. Two lists make two arrows.

So what does the score look like, when the lists are arrows?

The score tells you how the arrows sit against each other.

Leaning the same way: a positive score. The closer they line up, the bigger it gets.

At right angles: zero. Each arrow goes off in a direction the other does not care about.

Leaning opposite ways: a negative score.

One more thing changes the score: length. Make either arrow twice as long and the score doubles. A long arrow is a strong opinion. A short one is a weak one.

The score is big when two arrows point the same way and both are long. That is what “agree” means for two lists of numbers.

From scores to shares

One score is useful. Often you have several. Say you scored four routes, and you have one afternoon. You want to split your attention between them: most of it on the best match, a little on the next, nearly none on the worst.

So we need a way to turn scores into shares. Shares have two rules. None can be below zero. And together they must add up to exactly 100%.

The first idea is to divide each score by the total, as you would split a bill. Try it, and watch it break.

Plain dividing fails because scores can be negative, and a negative share means nothing. The fix has two steps.

Step one: raise. Turn every score into a positive number, in a way that keeps the order. The way that works is to multiply. Start from 1. For every point of score, multiply by about 2.7. For every point below zero, divide by about 2.7. A score of 2 becomes 2.7 × 2.7, about 7.4. A score of −1 becomes 1 ÷ 2.7, about 0.37. Small, but never zero and never negative.

Step two: divide. Now every number is positive, so splitting the bill works. Divide each raised number by the total.

You saw two things about this in the figure. Both matter later.

Only the gaps between scores count. Add 1 to every score and no share moves. Every raised number got 2.7 times bigger, and so did the total. What sets the shares is how far ahead one score is of another.

Bigger gaps make a sharper split. Double every score and the gaps double. The leader takes more. Push it far enough and the leader takes everything. Shrink all the scores towards zero and everyone gets an equal share.

Raise every score, then divide by the total. Out come shares: all positive, adding up to 100%, with the best score getting the most.

The two tools together

Now join them. Take one list that says what you are looking for. Score it against several other lists with the dot product. Then turn the scores into shares with softmax.

Below, the lists are words from chapter 6’s map, drawn as arrows. The orange arrow is yours to move.

Read the figure as a question and an answer. The orange arrow asks: “which of you is like this?” Each word answers with a score. Softmax turns the answers into a split of 100%.

Nothing in it was picked by hand except the orange arrow. Point it somewhere else and the shares re-form at once. One arrow, pointed the right way, is enough to pick out the words that matter and ignore the rest.

And notice what the length did. A short arrow spreads its shares evenly: it looks at everything a little. A long arrow puts nearly everything on one word. So a single list of numbers can say two things at once: what to look for, and how firmly.

What is still missing

You moved the orange arrow by hand. In a real machine, nobody does that. So where does the arrow come from? And once the shares are worked out, what are they used for?

You already have a hint for the second question. Shares are numbers that add up to 100%. Chapter 4 was about mixing things in set amounts. Shares are a ready-made set of amounts for a mix.

And chapter 6 left a problem open: the word “bat”, which needs to look at its neighbours to know what it means. A word that makes its own arrow, scores its neighbours, and mixes them by their shares would be able to do exactly that.

That machine is called attention. It is the next chapter, and you now have every part it is made of.