Attention mechanisms
How does a token decide what to read?
Turn the attention equation into an actual computation. Track every dimension, inspect a causal attention matrix, and read the same implementation that produced the interactive trace.
The article
Related reading
01
04
Before attention: how words become vectors
Build an embedding table, calculate a complete skip-gram training step, and discover why a decoder needs more than one fixed vector.
Why modern attention has so many names
Separate head sharing, position, compressed memory, sparse selection, and GPU execution. Then read DeepSeek’s recent designs with those distinctions in mind.