Why modern attention has so many names

1. One layer, several design decisions

Modern attention names describe different parts of a computation. Some change the stored representation, some choose which positions participate, and some change how the same mathematics runs on hardware. Understanding these axes makes the literature easier to read.

Start with the attention calculation and its query–key–value shapes. Then ask which decision a new method changes:

These choices can coexist. For example, DeepSeek-V3.2 applies sparse selection within an MLA-based architecture. A useful comparison identifies the changed component before comparing total memory or speed. [8]

2. MHA, MQA, and GQA organize sharing

Let \(H_q\) be the number of query heads and \(H_{kv}\) the number of key–value heads. Standard multi-head attention gives each query head its own K/V projections. Multi-query attention shares one K/V head across all queries. Grouped-query attention uses several groups, each sharing a K/V head. [1] [2] [3]

Shared keys do not force identical outputs: each query head still has its own query, hence its own attention weights. With contiguous grouping, query head \(i\) uses KV head \(\lfloor i/G\rfloor\), where \(G=H_q/H_{kv}\). Changing this architecture changes the model’s parameterization; it is not simply a runtime switch that preserves arbitrary pretrained MHA weights.

3. Inspect the grouping in twelve lines

This NumPy example has one batch item, four query heads, two KV heads, and dimension eight. A single decode query sees six allowed positions, so all provided keys are visible. The random arrays illustrate the tensor operation.

import numpy as np
rng = np.random.default_rng(7)
Q = rng.normal(size=(4, 1, 8))
K = rng.normal(size=(2, 6, 8))
V = rng.normal(size=(2, 6, 8))
Qg = Q.reshape(2, 2, 1, 8)
scores = np.einsum("hgtd,hsd->hgts", Qg, K) / np.sqrt(8)
scores -= scores.max(axis=-1, keepdims=True)
weights = np.exp(scores)
weights /= weights.sum(axis=-1, keepdims=True)
output = np.einsum("hgts,hsd->hgtd", weights, V).reshape(4, 1, 8)
print(output.shape)  # (4, 1, 8)

The reshape separates the head axis into two KV groups and two queries per group. In the first contraction, h identifies the shared KV head, g the query within its group, and s the key position. Softmax normalizes over s. The final contraction mixes the matching values and restores query-head order. No repeated K/V array is required. This calculation stops before the usual output projection.

4. Count the stored values

For ordinary K/V storage with equal key and value dimensions, the cache occupies:

The factor two counts keys and values; \(L\) counts layers, \(B\) batch items, \(T\) retained positions, and \(b\) bytes per element. Take two layers, one batch item, six positions, head dimension eight, and NumPy float64 at eight bytes per element:

These are calculated tensor-storage sizes, excluding weights, allocator overhead, and temporary buffers. Float64 matches the educational arrays above. Reducing bytes does not establish an equivalent reduction in wall-clock latency. See the KV cache walkthrough for how these tensors grow during generation.

5. RoPE makes position part of matching

Rotary position embedding applies position-dependent rotations to query and key coordinates. In the standard construction, their dot product contains a relative-position term:

Thus the score can depend on both content and positional separation. RoPE addresses how positions affect matching; head sharing addresses who owns K/V. The two choices can be combined. Extending supported context lengths involves additional model and training considerations beyond writing this identity. [4]

6. MLA learns a compact joint representation

DeepSeek-V2’s multi-head latent attention learns a joint latent representation for keys and values. Using column-vector notation, a simplified content path is:

The down-projection and reconstruction projections are learned within the architecture. This is different from applying a post-training SVD to an existing cache. In suitable inference algebra, projections can be absorbed into the query and output computations, allowing attention to use the compressed representation without materializing all expanded historical K/V.

Position introduces a complication: a position-dependent rotation prevents arbitrary projection absorption. DeepSeek-V2 therefore uses a separate RoPE key component. Its retained state includes both the joint latent \(c_t\) and the decoupled positional key. The cache is consequently larger than the content latent alone. MLA changes what is stored; it does not by itself select a sparse subset of positions. [5]

7. FlashAttention changes the execution strategy

FlashAttention computes exact attention by tiling operations and maintaining softmax statistics while reducing transfers between GPU memory levels. It avoids materializing the full attention matrix in high-bandwidth memory. “Exact” refers to the mathematical operation; floating-point evaluation order can still differ.

Full attention still involves a quadratic number of query–key interactions across a sequence. Better memory traffic does not make that interaction count linear. Kernel efficiency and architectural sparsity are separate questions. [6]

8. NSA and DSA choose context differently

Native Sparse Attention (NSA, 2025) combines three paths: compressed representations for broad context, selected token blocks for detailed information, and a local sliding window. Learned gates combine their outputs. Its design treats trainability and hardware-compatible access patterns as part of the attention mechanism. [7]

DeepSeek Sparse Attention (DSA, 2025) uses a lightweight “lightning indexer” to score past positions, then retrieves top-\(k\) entries for the main attention computation. In DeepSeek-V3.2, those entries belong to MLA. This is a distinct design from NSA’s three-path construction.

Count the selection work too. The DSA report describes an \(O(Tk)\) main attention computation across a sequence, while the indexer still performs \(O(T^2)\) scoring with cheaper operations. A fixed selection budget therefore does not make the entire architecture asymptotically linear. [8]

9. Research snapshot: 22 October 2026

The official DeepSeek-V4 report describes a hybrid of Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA compresses entries along the sequence dimension, then selects among them. HCA compresses more aggressively and attends densely over its compressed entries, with local context handled separately. Sequence compression and sparse selection are therefore different controls. [9]

The September 2026 DeepSeek-V4.1-Flash report extends this direction with CSA2 and cross-layer reuse:

Each mode still computes its own main query and local-window KV. The decoder’s hierarchical indexer lets later indexing layers search within a candidate pool built by the first Full layer. Their search cost is bounded for a fixed pool size; the initial Full layer still scans the visible range. [10]

Read these recent designs as specific combinations of storage, selection, and reuse choices. The mechanism descriptions here do not establish a universal accuracy or speed ranking.

Primary sources

1. Attention Is All You Need (2017).
2. Fast Transformer Decoding: One Write-Head is All You Need (2019).
3. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023).
4. RoFormer: Enhanced Transformer with Rotary Position Embedding (2021 preprint).
5. DeepSeek-V2 technical report (2024), §2.1.
6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022).
7. Native Sparse Attention (2025).
8. DeepSeek-V3.2 technical report (2025), §§2.1–2.3.
9. DeepSeek-V4 technical report (2026), §2.3; official model card.
10. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (17 September 2026), §2.3; official model card.