Jeonghyeon Kim
Under the Hood/Word representations

Word representations

How do words become useful vectors?

Start with the objects attention operates on. Learn how word vectors are trained, what a static lookup can express, and why sequence models need a way to retrieve relevant context.

Read the full explanation →

The article

01

Before attention: how words become vectors

Build an embedding table, calculate a complete skip-gram training step, and discover why a decoder needs more than one fixed vector.

14 min readEquations and code

Related reading

02

Inside attention, one tensor at a time

Follow a complete decoder through Q, K, V, rotary positions, grouped-query attention, residual connections, and next-token logits. Every shape has a job.

22 min readEquations and code