Words as places: word embeddings

Every machine so far has eaten numbers. To read, a machine needs words turned into numbers. This chapter finds the right way to do that: give every word a place on a map, where words that are alike sit close together.

By the end of this chapter, you will be able to

Say why simply numbering the words does not work.
Turn a word into a place: a short list of numbers, where near means alike.
Explain how a machine finds those places by itself, from nothing but text.
Do a sum with words: take a step from one word and land on another.
Say what an embedding is.

The first thing anyone tries

Chapter 5 ended with a question. What is 0.5 times “bicycle”? It has no answer. Knobs multiply numbers, so a word has to become a number before a machine can touch it.

The easy way is to number the words. Put them in A to Z order and count: apple is 1, bus is 2, cake is 3. Now every word is a number. Is that enough?

Try it. Below, a machine has to learn one simple thing about eight words: can you eat it? The orange line is the shape it must learn. Chapter 5 showed that every bend in a shape costs the machine a neuron.

Try itThe words are numbered in A to Z order. The orange line is the shape a machine must learn to answer “can you eat it?”. Tap two words to swap their numbers. Make the shape as simple as you can.

6 bends

This line has 6 bends. It goes up and down because the numbers say nothing about the words. “Cake” sits between “bus” and “cat” only because of its spelling.

The A to Z numbers were not wrong. They were empty. They said “cake” is between “bus” and “cat”, which is true of the spelling and of nothing else. So the machine had to learn every word one at a time, with a bend for each.

When you moved the foods together, something better happened than a simpler line. Now what the machine learns about “cake” spills over to “rice”, because they are neighbours. A new food, placed next to them, would get the right answer without any more learning.

A word’s number should say something about the word. Words that are alike should get numbers that are close.

One number is not enough

So let us pick numbers that mean something. Take some animals, and give each one a number for its size. Small animals get small numbers. Big animals get big ones.

That is already better than A to Z. But a single line of numbers runs out of room fast. Words are alike in more than one way.

Try itEach animal sits on a line by its size. On this line a cow and a tiger are almost the same thing. Give each word a second number: drag the wild animals up, and leave the tame ones low.

One number each. By size alone, the nearest word to cow is tiger. A machine reading these numbers would treat them as nearly the same animal.

On the line, a cow was next to a tiger, and there was no way to fix it. Move the tiger away from the cow along the line, and it bumps into something else. A line only has one direction to be different in.

The second number gave the words a new direction. Now the tiger can be close to the cow in size and far from it in wildness, both at the same time.

A word is no longer a number. It is a place: a short list of numbers, one for each direction. And “near” on the map means “alike”.

In symbols

A place is written as a list of numbers in brackets. Say you left the tiger high up, like this:

tiger = (7.0, 8.5)

The first number is how far along. The second is how far up. The order matters.

The figure kept telling you which word was “nearest”. To know that, it measured the straight-line distance between two places. That is the rule about right-angled triangles you may know from school:

distance (a, b) = (a_{1} - b_{1})^{2} + (a_{2} - b_{2})^{2}

Take the gap in the first number and the gap in the second. Square both, add them, and take the square root.

$a,; b$: two places, such as tiger and bear.
$a_{1},; a_{2}$: the first and second number of place $a$ .

With tiger at $(7.0, 8.5)$ and bear at $(8.8, 8.0)$ , the distance is $1. 8^{2} + 0. 5^{2} \approx 1.9$ . Squaring the gaps is the same trick as in chapter 2: it makes every gap count as positive.

Who decides where the words go?

You placed eight animals by hand, using two things you already knew about them. That does not go far. A language has tens of thousands of words. And which directions should there be? Size and wildness are fine for animals. They say nothing useful about “Tuesday” or “because”.

You have met this problem before. In chapter 1 you set a knob by hand. Then the article spent two chapters teaching a machine to set it for you. The same move works here.

Make the place of every word a set of knobs. Then let the machine turn them.

To learn, a machine needs a job to be wrong at. Here is a job that needs nothing but text. Take a sentence, cut out one word, and ask: which words could go in the gap?

The machine below answers with parts you know. Each gap has a weighted sum, like chapter 4. What goes into the sum is the two numbers of a word’s place. A big answer means “yes, this word fits here”. At the start, all the knobs are a jumble: the gaps’ knobs, and the words’ places too.

Try itThe machine gets 23 short sentences and twelve words dropped in a jumble. Its job: say which words fit which gaps. Press the button and watch where the words go.

typical miss0.54

the text

Tap a sentence to see which words were seen in its gap.

step 0 of 600

Before reading, the places mean nothing. Foods, animals and vehicles are all mixed up. A typical miss is 0.54.

Why did the words sort themselves? Think about what a weighted sum does with a place. Two words at the same spot feed it the same two numbers. So they get the same answer, for every gap.

“Cat” and “dog” need the same answers. Both fit after “I fed the”. Neither fits after “I parked the”. The simplest way for the machine to give them the same answers is to put them in the same place. “Cat” and “bus” need different answers, so they have to move apart.

Nobody gave the machine a list of animals. Nobody named the directions on its map. Words that turn up in the same kinds of sentences ended up in the same part of the map. That is all the machine knows about meaning, and it turns out to be a lot.

And the learning itself was nothing new. It was the loop from chapter 3, with the blame passed backwards as in chapter 5. The blame just travelled one hop further than before: through the gap’s knobs, and into the word’s own two numbers.

One honest detail. In the figure, cat, dog and cow landed on exactly the same spot. That is correct. In this tiny text, they appear in exactly the same sentences, so the machine has no reason to tell them apart. Give it a sentence about milking, and the cow would move.

In symbols

The rule for one gap is chapter 4’s weighted sum. The only change is what goes in.

y^= u_{1} e_{1} + u_{2} e_{2} + c

How sure the machine is that the word fits the gap. Close to 1 means yes. Close to 0 means no.

$e_{1}, e_{2}$: the two numbers of the word’s place. They are orange now, because they are knobs.
$u_{1}, u_{2}$: the gap’s two knobs: how much it cares about each direction of the map.
$c$: the gap’s bias.

In chapter 4, the things going in were measured: kilometres, hills. Here, both halves of every product are knobs. The machine learns what to look for, and it also learns what it is looking at.

\frac{\partial L}{\partial e _{1}} = 2 (y^- y) \times u_{1}

The blame for a word’s first number, from one gap: the miss, times the knob on its wire. It is the same chain as in chapter 5. A word gets blame like this from every gap, and they are added up.

Steps on the map mean something

If places carry meaning, then the step from one place to another should carry meaning too. Going from “kitten” to “cat” is a step in some direction. What happens if you take the same step from “puppy”?

A step is easy to work out. Take the place you end at, and subtract the place you start from, number by number. To take a step, add it on, number by number.

Try itTap two words to draw a step from the first to the second. Then tap a third word, and take the very same step from there. Start with kitten, then cat, then puppy.

Eight words, each at its place. Tap kitten to begin.

The step from kitten to cat does not mean “cat”. It means “grow up”. Add it to a puppy and you get a dog. Add it to a calf and you get a cow. Try the step from kitten to puppy as well. That one means something like “the same age, but a dog”.

On a good map, a direction is a meaning. That is how you do arithmetic on words: you add and subtract their places.

One honest note. This small map was laid out by hand, to make the idea easy to see. But the same thing shows up on maps that machines learn from billions of sentences. A well-known case: start at “king”, take away “man”, add “woman”, and you land close to “queen”. It is never exact, and it does not work for every word. It still says something striking: the machine was never told about growing up, or about kings. It only filled in gaps.

More than two numbers

Two numbers made a map you can draw. But two directions are still very few. On our animal map there is no room to say that a cat purrs, or that a horse can be ridden, or that a cow gives milk.

So real machines give every word a longer list. Not two numbers, but hundreds, or thousands. Nobody can draw a map with a thousand directions. Nothing else changes. A word is still a place. Near still means alike. A step is still a meaning. And the distance rule and the weighted sum work the same, with more terms to add.

More directions simply means more ways to be alike at once. “Horse” can be near “cow” in one direction, near “bike” in another, and near “gallop” in a third.

What is still missing

Words are now places, and a machine can find the places by itself. Two things are still missing.

First, look again at that weighted sum. It takes two lists of numbers, multiplies them pair by pair, and adds up. The answer was big when the word suited the gap. So this small sum is doing an important job: it measures how well two lists agree. It deserves a closer look, and a name. That is the next chapter.

Second, every word has one place, fixed for good. But think of “bat”. In “the bat flew out of the cave” it is an animal. In “she swung the bat” it is a piece of wood. One pin on the map cannot be in two places. To know which “bat” it is, the word would have to look at the words around it.

A way for a word to look at its neighbours, and to choose which ones matter, is called attention. We are two chapters away.

You can now

Say why simply numbering the words does not work.
Turn a word into a place: a short list of numbers, where near means alike.
Explain how a machine finds those places by itself, from nothing but text.
Do a sum with words: take a step from one word and land on another.
Say what an embedding is.