Under the Hood
Notes on how language models work.
From word vectors to attention and efficient inference, with the equations, tensor shapes, and code worked out along the way. If you are starting from scratch, read the topics in order.
Contents
Word representations
From a row in an embedding table to a representation that depends on context.
Attention mechanisms
Queries, keys, values, and the exact path through a small Llama-style decoder.
KV cache
Follow prefill and decoding. See which tensors stay, which grow, and why.
Modern architectures
Understand GQA, latent memory, sparse attention, and efficient GPU kernels.
How we got here
- 2003–2013
Learned word representations and word2vec: words become reusable vectors. Start with embeddings.
- 2014–2015
Encoder–decoder attention learns to retrieve source states while generating a translation. The fixed-vector bottleneck.
- 2017
The Transformer makes attention a central operation in sequence modeling. Work through Q, K, and V.
- 2019–2024
MQA, GQA, FlashAttention, and MLA address different costs of long-context inference. Compare the mechanisms.
- 2025–2026
Sparse selection, compressed context, and cross-layer reuse extend the design space. Read the recent work.
Code and experiments
- NumPy decoder
Two layers, four query heads, two KV heads. A complete forward pass you can read and run.
- Line-by-line walkthrough
Follow the implementation alongside the shapes it computes.
- Measured attention and cache trace
The values behind the interactive figures, including numerical checks.
- Pretrained TinyLlama experiment
A prepared GPU probe for the 1.1B checkpoint. The GPU run is still pending.
The NumPy example uses deterministic random weights to make the mechanics easy to inspect. The pretrained experiment is separate.
Last updated 22 October 2026. Primary references are linked in each article.