How Do Word Embeddings Capture Relationships Between Words?

Welcome back!

Let me ask you something.

Do you walk every day? or exercise every day?

Our lives have become increasingly busy, and our lifestyles have changed quite a lot compared to earlier times.

But, we still try to find some time to walk or exercise regularly to stay fit and healthy.

I also decided to make running a part of my daily routine.

When we think about walking or running, one thing that comes to mind is shoes.

I also wanted to buy a good pair of shoes for running.

And if something is available online, we often prefer shopping online because we can take our time to compare different products and choose what we want.

So, I decided to buy my shoes online too.

I opened my shopping app and searched for “running shoes”.

I came across various shoes from different companies, but I noticed something interesting.

Many shoes were described as running shoes, while other shoes were described as walking shoes, sports shoes, sports sneakers, and so on.

But I had searched only for “running shoes.”

So, one question came to my mind:

How does the app know that “running shoes” and “sports shoes” are related, even though they are different words?

While trying to understand this, i came across a concept called word embeddings.

···

The Shopping Is Done. Now Let’s Understand What’s Happening.

Before starting our journey, there is one more thing I want to mention: the significance of embeddings in modern AI.

Large Language Models (LLMs) also use embeddings as part of processing their input, such as our prompts.

You may also have noticed that AI assistants can sometimes retrieve relevant information from the web or other sources and use it to generate a response.

This approach is commonly known as Retrieval-Augmented Generation (RAG).

Of course, modern search systems may combine different techniques, such as keyword matching, embeddings, and ranking models.

However, embeddings play a significant role in many modern AI applications by helping systems identify semantic relationships and retrieve information based on meaning, rather than just depending only exact word matches.

···

What Are Word Embeddings, and Why Do We Need Them?

In simple words, an embedding is a numerical representation of something.

In the case of word embeddings, a word is represented as a vector of numbers.

You might get a doubt that we already did this in the previous blog on TF-IDF.

Let’s look at what we did with TF-IDF in the last blog.

We considered a few sentences and wanted to classify them as positive or negative.

For that, we converted the sentences into a numerical representation using the TF-IDF method.

But we were representing the text based on the importance of the words in those documents, rather than learning a separate representation for each word based on its context.

If we look at TF-IDF, it is mainly a statistical method. It depends on things like how frequently a term occurs in a document and how common or rare that term is across the collection of documents.

But in the case of word embeddings, things are different.

Here, our goal is to learn a vector representation for each word based on how the word is used with other words.

We are trying to learn relationships between words from the data.

···

How Can a Model Learn Relationships Between Words?

What do we mean by a learned vector representation? How does the learning happen?

Let’s understand this clearly using a simple example.

Let’s say we want to build a search system for an online shopping website.

As we discussed earlier, if we search for running shoes, there might be products in the same catalog that are described as jogging shoes, sports shoes, or something similar.

If the system only looks for exact word matches, we might miss some relevant products.

So, we want the system to learn that words that appear in similar contexts can be related.

But we don’t want to manually create a dictionary saying which words are related and which are not.

Instead, we want the system to learn these relationships from the text itself.

···

To understand how the learning process works, let’s consider a few sentences.

1. The tasty food was served.

2. I enjoyed the tasty meal.

3. The restaurant serves delicious food.

4. The new car has a powerful engine.

5. The large truck has a powerful engine.

Now we have 5 sentences. This is our simple dataset.

Our goal here is to understand how embeddings are learned, i.e., how a word is represented as a vector based on the context in which it appears.

Here, we want to understand the detailed math behind word embeddings, so we are using a simple dataset.

Now, by looking at these five sentences, what can we say?

We can recognize that food and meal are related.

Similarly, from the last two sentences, we can recognize that car and truck are related as they also appear in similar contexts.

But the machine doesn’t automatically know these relationships.

We need the machine to learn these relationships from the text.

First, let’s have a look at the unique words in our dataset.

After converting all the words to lowercase and removing the duplicate words, we got 19 unique words in our vocabulary.

Now, to find relationships between words, we need to represent each word numerically.

This means we need a vector representation for each word in our vocabulary.

···

How Does Word2Vec Learn from Context?

One popular technique for learning word embeddings is Word2Vec.

It has two architectures: one is Skip-gram, and the other is CBOW (Continuous Bag Of Words).

In this blog, let’s first focus on Skip-gram.

To understand how Skip-gram works, let’s consider a sentence from our dataset:

‘The tasty food was served.‘

Next we select a word from the sentence as a ‘target word‘.

Let’s choose ‘food‘ as our target word.

Next, we select a context window.

For example, if we set the context window to 1, we consider one word on either side of the target word. We call these words context words.

Here, let’s set the context window to 1.

Then, we have:

For the target word ‘food’, we have context words ‘tasty’ and ‘was’.

Here, by “target word,” we mean that we are trying to learn a numerical representation for the word food. To do this, first we look at the context in which the target word occurs, using a context window.

In our example, the target word food occurs with two context words tasty and was.

From this, we can create two target-context pairs: (food,tasty) and (food,was).

Now these pairs serve as the training examples for our Skip-gram model.

···

Now the question is, what does Skip-gram actually do with these pairs?

Skip-gram takes the target word and tries to predict the context words that appear around it.

For example, we have a pair (food,tasty). We give food to the model and ask it to predict tasty.

By doing this, the model learns from patterns in the text about which words tend to occur around each other.

This might be confusing at first, but it will become clearer once we work through an example.

But how Skip-gram takes the target word and predicts the context word.

Let’s see.

How Are Words Represented as Vectors?

Here, we want to find the embedding for each word, which is a learned numerical representation of that word.

We have 19 unique words in our vocabulary, which means we need to learn an embedding for each word.

But first, we need to decide the dimension of the embeddings.

If we select the embedding dimension as 2, then each word will be represented by a 2-dimensional vector.

For example, the initial embedding for food looks like this:

At the start of training, we initialize the embedding values for each word with random values.

These values will later be updated during training.

Now, we have 19 unique words, and each word has a 2-dimensional embedding. Therefore, the shape of our embedding matrix be:

W is the embedding matrix and R denotes the set of real numbers.

It looks like this:

As we initialize the embeddings for all the words, the model needs a way to know which word corresponds to which row of the embedding matrix.

For this, we can represent each word using a one-hot vector based on its position in the vocabulary.

For example, consider food , which is our target word for now,

Its one-hot representation is: [0,0,1,0,…….,0]

The 1 tells us the position corresponding to food in our vocabulary.

The model can then use this position to retrieve the corresponding row from the embedding matrix.

So, if the third row of our embedding matrix is: [0.2,  0.7]

then the model retrieves:

This process of retrieving the vector corresponding to a word from the embedding matrix is called as embedding lookup.

In practice, implementations usually use the word’s integer ID to retrieve the corresponding row directly, instead of constructing a full one-hot vector.

···

Embedding Dimension

Before proceeding further, let’s first discuss embedding dimension.

In our example, we selected the embedding dimension as 2 because it keeps the mathematics simple and will be easy for us to understand the process.

But in real-world applications, embedding dimensions can be much larger.

For example, dimensions such as 100, 300, 512, 768, 1024, 1536, and even larger values can be used, depending on the model and application.

Let’s say we selected an embedding dimension of 300.

Now, what does an embedding dimension of 300 actually mean?

We know it already that, each word now is represented by a 300-dimensional vector containing 300 numerical values.

Mathematically, we can think of it as a point in a space with 300 dimensions or 300 coordinate axes.

Here, one question comes to the mind.

What do those 300 axes actually represent?

The dimensions of a learned word embedding usually do not have a directly interpretable meaning for humans.

We don’t normally have something like:

Dimension 1 = “food-relatedness”

Dimension 2 = “emotion”

Dimension 3 = “size” etc.

For example, let’s say our vocabulary contains 10,000 words.

In the large text corpus, we may have patterns related to syntax, semantic relationships, topics and word usage.

This information can be distributed across many dimensions, and the model learns the numerical values of these dimensions from the patterns in the training data.

How Do We Choose the Embedding Dimension?

The embedding dimension is a hyperparameter.

In practice, we can experiment with different values and evaluate how the resulting embeddings perform for the task we want to accomplish.

The choice can also depend on factors such as the size of the training data, computational resources, and the complexity of the problem.

So, there isn’t one universally “correct” embedding dimension.

A larger dimension provides more capacity to represent patterns, but it also increases the number of parameters, memory requirements and computational cost.

We’ll get a better understanding of how embedding dimensions are chosen as we move into more advanced concepts.

···

Now after selecting the embedding dimension, we initialize the embeddings with random values for every word in the vocabulary.

This means we initialize 19 target word embeddings, one for each word in our vocabulary.

During training, each word can become a target word, and its embedding is updated based on the training examples.

Let’s see how it is done.

Here, we selected the embedding dimension as 2, and now let’s focus on one word which is the current target word, food.

Let’s initialize the embedding for food.

For food we already have two target-context pairs: (food,tasty) and (food,was).

Here we might get confused between context window and embedding dimension. They are two different things.

The context window determines which nearby words are considered context words, while the embedding dimension determines how many numerical values are used to represent each word.

We’ve initialized the embedding for the target word food, and now we have two context pairs.

Now what we should do?

Let’s just consider a one target-context pair: (food,tasty)

We want the model to learn that tasty is a context word for food.

What does that mean?

We already have a numerical representation for food.

Why Does Word2Vec Use Two Sets of Vectors?

Now, the model needs a way to represent tasty as a possible context word so that it can learn that food and tasty are associated in this context.

For this, we need a context/output vector for tasty.

Just as we initialize target/input embeddings for all words in the vocabulary, we also initialize context/output vectors for all words in the vocabulary with random values.

So, at the beginning of the Skip-gram training process, we have two sets of vectors:

19 target/input vectors, one for each word in our vocabulary and

19 context/output vectors, one for each word in our vocabulary.

In total, our simplified example has 38 vectors.

We can think of Skip-gram as working with these two different sets of vectors during training.

···

Now, let’s look at the table showing the initialized word vectors for the target/input words and context/output words.

Now, for our target-context pair (food,tasty), we have the initialized target/input vector for food and initialized context/output vector for tasty.

We denote the target/input vector by v and the context/output vector by u.

···

How Does Word2Vec Score a Target-Context Pair?

Now, simply having these vectors is not enough.

We need to measure how strongly the current target vector and context vector support this particular target-context pair.

This is where we use the dot product.

We already know that vectors pointing in similar directions tend to have a larger dot product, whereas vectors pointing in different directions tend to have a smaller or negative dot product.

Let’s calculate the dot product:

We now have a score of 0.64 for the pair (food,tasty).

Why Isn’t a Dot Product Enough?

But this score alone is not enough for the model to determine whether tasty is the appropriate context word for food.

We need to compare it with other scores to know how it stands relative to them.

As our vocabulary contains 19 words, we need to calculate the dot product between the target vector food and the context/output vector of each word in the vocabulary.

This gives us 19 scores, one for each possible context word, which we can compare to determine which words the model currently considers more likely to appear in the context of food.

Now, we have calculated the dot-product scores for all 19 words in our vocabulary.

But how can we use these scores to help the model learn useful word representations?

For our target-context pair (food, tasty), we want our model to learn to predict tasty when it receives food as the target word.

However, the model’s vectors have been randomly initialized, so it hasn’t yet learned meaningful relationships between words.

If we look at the dot-product scores, we can observe that tasty received the highest score.

But this is purely coincidental because these scores were calculated using randomly initialized vectors.

We shouldn’t assume that the model already knows that tasty is the correct context word for food.

At this point, we might think that we should train the model to increase the dot-product score for tasty.

But there’s something important to consider.

The target vector for food is used to calculate scores for all candidate context words.

During training, when we update this target vector, its dot products with multiple context vectors may change.

We might think, So what if the other dot-product scores increase or decrease? We only want to increase the score for tasty, right?

However, simply increasing the score of the correct word doesn’t tell us whether the model has learned to distinguish it better from the alternatives.

We want the model to assign a higher probability to tasty given food, relative to the other context words.

This is where we need probabilities.

Probabilities help us understand how the model distributes its predictions across the context words.

For our target-context pair (food, tasty), we want the model to assign a higher probability to tasty, because it is the correct context word for this example.

But how do we convert the dot-product scores into probabilities?

This is where the Softmax function comes in.

How Does Softmax Turn Scores into Probabilities?

Softmax converts the dot product scores into probabilities between 0 and 1, with all probabilities adding up to 1.

Here’s the formula:

where

As we have calculated the dot-product score of 0.64 for the pair (food,tasty), let’s convert it into a probability using the Softmax function.

We got a probability of approximately 6.74% for the word tasty, given the target word food.

This probability is based on our randomly initialized vectors.

The model has not yet learned meaningful relationships from our dataset.

But there’s something important for us to understand here.

Softmax considers all the words in the vocabulary when calculating the probability of a candidate word.

Even if the dot product for tasty remains unchanged, its probability can change if the scores of the other candidate words change.

This is because the denominator included the scores of all 19 words.

In this way, Softmax converts the dot-product scores into a probability distribution over the vocabulary.

For our target-context example (food,tasty), we want the model to assign a higher probability to tasty, since it is the actual context word.

As the model learns from more training examples, it adjusts its vectors to improve its predictions.

But how does the model learn to adjust these vectors?

We’ll explore this in the next part.

···

I hope you have learned something from this blog to get started with word embeddings.

Now, we have reached a point where we can begin to explore how embeddings are actually learned.

There is still a lot to learn, but we will explore everything step by step to build a good understanding of how this process works.

See you in Part 2!

If you haven’t read my recent article on TF-IDF, you can read it here.

If anything isn’t clear, feel free to let me know in the comments on LinkedIn.

I’d love to hear your feedback!

Thanks for reading!