Before attention: how words become vectors

1. Start with the representation

Before asking how attention connects tokens, ask what a token looks like to a neural network. A word such as cat must become a numerical object. The important questions are how that object is learned, what information it contains, and whether its meaning changes with the sentence.

This lesson follows the path from a vocabulary index to a learned word vector, then to a representation that depends on context. Word2vec is a useful starting point because its prediction task is small enough to calculate by hand. It belongs to the history of representation learning; attention enters when we need a flexible way for positions to exchange information.

Learned word representations predate word2vec. Bengio and colleagues jointly trained distributed word representations and a neural language model in 2003. Their central motivation was generalization: similar representations can help a model handle word combinations it has never seen. [1]

2. A one-hot vector selects a trainable row

Use the toy vocabulary ["the", "cat", "drinks", "milk"]. A one-hot encoding of cat is \(x_{\mathrm{cat}}=[0,1,0,0]\). It identifies the word, but its geometry supplies no semantic relationship: any two different one-hot vectors have dot product zero and Euclidean distance \(\sqrt{2}\).

Now introduce an embedding table. With vocabulary size \(V\) and embedding dimension \(d\), the shapes are:

Matrix multiplication here is simply a lookup: \(x_wE_{\mathrm{in}}=E_{\mathrm{in}}[w]\). Implementations can retrieve the row directly instead of constructing the one-hot vector. The vocabulary index is fixed; the row values are learned. A compact vector becomes useful through its training objective, rather than through an inherent meaning assigned to each coordinate.

A model that takes one-hot inputs can therefore learn relationships through its embedding table. The limitation concerns the original one-hot geometry, not everything a model can learn from it.

3. Word2vec learns from neighboring words

Mikolov and colleagues proposed two efficient architectures in 2013. Continuous bag of words (CBOW) combines surrounding word vectors to predict a target word. Skip-gram uses a center word to predict nearby words. In the sentence “the cat drinks milk,” a context window of one produces the skip-gram pairs (cat, the) and (cat, drinks). [2]

The learning signal comes from observed co-occurrence. Across many examples, words with similar predictive relationships can acquire related representations. Similarity is consequently shaped by the corpus and objective; it is not a guarantee that nearby words are synonyms.

The full-softmax version uses two tables: an input vector \(v_c\) for the center word and an output vector \(u_o\) for a candidate context word. Their dot product supplies a compatibility score. In the dot-product formulas below, individual retrieved embeddings are written as column vectors. The conditional probability and single-pair loss are: [3]

Both tables contain parameters that training can change. Using two tables lets a word play distinct roles as a predictor and as a prediction target. In the following NumPy calculation, each stored row is treated as a one-dimensional vector.

4. Calculate one complete training step

This standalone CPU example uses four words and two-dimensional vectors. Every starting coordinate is illustrative. These numbers are chosen to make the arithmetic visible; they are not measurements from a trained language model.

import numpy as np

vocabulary = ["the", "cat", "drinks", "milk"]
word_to_id = {word: i for i, word in enumerate(vocabulary)}

E_in = np.array([
    [0.1, 0.0], [1.0, 0.0], [0.0, 1.0], [1.0, 1.0]
])
E_out = np.array([
    [0.0, 0.0], [1.0, 0.0], [0.0, 1.0], [1.0, 1.0]
])

def softmax(scores):
    shifted = scores - scores.max()
    weights = np.exp(shifted)
    return weights / weights.sum()

center_id = word_to_id["cat"]
context_id = word_to_id["drinks"]
v = E_in[center_id].copy()

logits = E_out @ v
probabilities = softmax(logits)
loss = -np.log(probabilities[context_id])
print("logits:", logits)
print("probabilities:", np.round(probabilities, 6))
print(f"loss before: {loss:.6f}")

grad_logits = probabilities.copy()
grad_logits[context_id] -= 1.0
grad_v = E_out.T @ grad_logits
grad_out = np.outer(grad_logits, v)

learning_rate = 0.1
E_in[center_id] -= learning_rate * grad_v
E_out -= learning_rate * grad_out

new_probabilities = softmax(E_out @ E_in[center_id])
new_loss = -np.log(new_probabilities[context_id])
print("new center vector:", np.round(E_in[center_id], 6))
print(f"loss after: {new_loss:.6f}")

Running this example produces:

logits: [0. 1. 0. 1.]
probabilities: [0.134471 0.365529 0.134471 0.365529]
loss before: 2.006409
new center vector: [0.926894 0.05    ]
loss after: 1.835051

Read the calculation in six steps:

1. Select the center vector. E_in[center_id] retrieves the row for cat, initially \([1,0]\). The copy preserves that forward-pass value while parameters are updated later.
2. Score every candidate. E_out @ v multiplies a \(4\times2\) table by a length-2 vector. The resulting four logits correspond to the vocabulary order.
3. Normalize the scores. Softmax produces a probability distribution. Subtracting the largest score leaves the mathematical probabilities unchanged and improves numerical stability.
4. Measure the target error. The observed context word is drinks. Its probability is approximately \(0.134471\), so the loss is \(-\log(0.134471)\approx2.006409\).
5. Differentiate the loss. For softmax cross-entropy, the logit gradient is \(p-y\), where \(y\) is the target one-hot vector. The matrix product and outer product propagate that gradient to the two embedding tables.
6. Move the parameters. Subtracting the gradients changes the center row and the output rows. We compute both gradients before changing either table, so they refer to the same forward pass.

The lower loss verifies this particular update on this particular pair. It does not establish useful language representations. That requires many examples, appropriate optimization, and evaluation beyond a single training pair.

5. Negative sampling changes the objective

Full softmax scores the entire output vocabulary for every training pair. The later 2013 word2vec paper introduced negative sampling as an efficient alternative: distinguish an observed pair from pairs formed using sampled noise words. For negatives \(n_1,\ldots,n_k\), a sampled loss is: [3]

This objective rewards a high score for the observed pair and low scores for sampled pairs. It trains binary discrimination problems. Its sigmoid scores do not directly constitute a normalized probability distribution over all vocabulary words. Keep that distinction when comparing implementations: full softmax and negative sampling are different objectives, even though both can learn useful embeddings.

6. One word type can have several meanings

Consider “She sat by the river bank” and “She called the central bank.” A standard trained word2vec table retrieves the same row for bank in both sentences. The training used contexts, but the final lookup depends on the word identity alone.

Static describes this context-independent lookup; it does not mean the vectors were never trained. A contextual model instead computes a representation such as \(h_i=f(x_1,\ldots,x_T,i)\), which can vary with the surrounding sequence. For a causal model, the available context is limited to positions up to \(i\).

Contextualization also does not require attention. ELMo used internal states of a bidirectional language model to produce context-sensitive representations. BERT later used bidirectional Transformer representations. These are different architectures pursuing a related representational goal. [4] [5]

Even a contextual model may begin with a static token-embedding lookup. Subsequent layers transform that initial vector using the available context. The input embedding and the final contextual representation are different stages of the computation.

7. From word vectors to a whole sequence

An RNN processes embeddings sequentially, updating a hidden state:

The state can incorporate information from the preceding tokens. In early encoder–decoder designs, the encoder represented the input sequence with a fixed-size state, and a decoder used that representation to generate an output sequence. Cho and colleagues described this structure in their June 2014 encoder–decoder paper. [6]

The potential bottleneck is easy to picture in translation: all source information must remain accessible through that fixed-size representation as the decoder produces different target words. Longer or more complicated inputs may increase the demand on it. This is a capacity and optimization concern, not a theorem that a fixed-size vector must fail on every long sequence.

Bahdanau and colleagues made the bottleneck explicit and let the decoder use different weighted combinations of encoder states at different output steps. Their preprint appeared in September 2014 and was presented at ICLR 2015. [7]

8. The next question: which information is useful now?

Suppose the encoder keeps states \(h_1,\ldots,h_T\). Attention calculates relevance scores for the current decoder step, normalizes them into weights, and forms a context vector:

The decoder can retrieve a different mixture at every step. In Bahdanau attention, a learned alignment function uses the decoder state and each encoder state to obtain the scores. [7]

This establishes the next lesson’s central problem: how to compute relevance, and how to turn the resulting weights into useful information. From there, query–key–value notation, self-attention, multi-head attention, and efficient attention architectures become concrete extensions of operations we can inspect.

References

1. Bengio et al. (2003). A Neural Probabilistic Language Model.
2. Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space.
3. Mikolov et al. (2013). Distributed Representations of Words and Phrases and their Compositionality.
4. Peters et al. (2018). Deep Contextualized Word Representations.
5. Devlin et al. (2018 preprint; 2019 proceedings). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
6. Cho et al. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.
7. Bahdanau et al. (2014 preprint; ICLR 2015). Neural Machine Translation by Jointly Learning to Align and Translate.