Inside DeepSeek-V4.

Most people, when told that LLMs use "attention," picture a square grid where every word looks at every other word. This was true in 2017. It is no longer how a frontier model handles a million-token context, because the cost of that square grid is, in a literal sense, quadratic.

DeepSeek-V4 interleaves two layer types. Compressed Sparse Attention (CSA) takes every four tokens, crushes them into a single entry, and lets a small "lightning indexer" pick out the top thousand or so entries that matter for each query. Heavily Compressed Attention (HCA) compresses one hundred and twenty-eight tokens into a single entry and attends to all of them densely, because by then there are very few. Both layer types are wrapped around a small sliding window of recent uncompressed tokens, so local detail is never lost.

The cleanest mental image is a zoom-out map. Close to the query, individual streets. A bit further, neighborhoods. The whole rest of the city, drawn as labelled districts. The query never has to look at every street.

Every token a model sees in a conversation, by default, is stored forever. Its key and value projections sit in memory, in sixteen-bit precision, ready to be attended to by every token that comes later. This is what makes long-context expensive. It is also why a chat that once ran in a few hundred megabytes can, ten thousand turns in, demand fifty gigabytes per worker.

DeepSeek-V4 attacks the problem from three sides at once. The attention layers compress what they store: four tokens to one in CSA layers, a hundred and twenty-eight to one in HCA layers. The precision drops too. Main key-value entries live in FP8, only the rotary positional dimensions need BF16, and the indexer's QK path runs in FP4. And the layers themselves are interleaved so most of the depth pays the heavily-compressed cost rather than the moderate one.

The result, at a one-million-token context: roughly two percent of the standard sixteen-bit GQA8 baseline.

The chart's punchline is the y-axis. Three of the four lines crowd into the lowest few percent of the plot area. The textbook "every token costs another sixteen-bit key-value pair" line shoots diagonally to the top right. Everything else stays close to the floor.

The residual connection is the most quietly load-bearing idea in modern deep learning. It says: at each layer, instead of replacing the running state, simply add to it. This is the reason a hundred-layer network trains at all. It is also, everywhere, the same single addition.

DeepSeek-V4 widens the running state into four parallel streams and lets each layer mix them through a learned 4×4 matrix. Plain hyper-connections, just any old 4×4 mixing matrix, train unstably. The signal blows up or collapses across hundreds of layers. The fix is to constrain that matrix to live on a mathematical object called the Birkhoff polytope: the set of doubly-stochastic matrices whose rows and columns each sum to one. Twenty steps of a Sinkhorn-Knopp iteration project the learned matrix onto that manifold, every forward pass.

The intuition is plumbing. A single residual stream is one river flowing through the network. Hyper-connections turn it into four braided rivers. The doubly-stochastic constraint is the assurance that no layer accidentally dumps all the water into one channel and dries the other three.

AdamW has been the default optimizer for transformers since GPT-2. Its update rule is, to a first approximation, a normalized version of the gradient. If the gradient is stretched in one direction, the step is stretched in that direction. This is fine for moderate-sized models. At trillion-parameter scale, those stretches turn into instability.

Muon takes the gradient matrix M, computes its singular value decomposition M = UΣVᵀ, throws away Σ, and steps in the direction of UVᵀ. The step has the same direction structure as the gradient, but every singular value is now 1. Real SVD is too expensive to do per-step, so DeepSeek-V4 uses a hybrid Newton-Schulz iteration (ten steps in two stages) that approximates it cheaply.

The geometric picture: AdamW takes a stretched ellipse and steps along its longest axis. Muon turns that ellipse into a sphere, then steps. Same direction, different magnitude. The result is faster convergence and better stability at scale. AdamW is still used for the bits where matrix orthogonalization doesn't make sense: embeddings, the prediction head, and RMSNorm weights.

The math people quote about LLM size, "175 billion parameters times two bytes equals 350 gigabytes," is the math of a model stored in 16-bit floats. That has been the working assumption since 2018. It is the reason frontier models do not fit on single GPUs without tricks.

DeepSeek-V4 trains in mixed precision from the start. The Mixture-of-Experts expert weights live in FP4 (four bits) with quantization-aware training so the model adapts to the precision loss instead of being post-hoc compressed and hoping. The attention indexer's QK path also runs in FP4. The main KV entries live in FP8; only the rotary positional embedding dimensions need BF16. Index scores quantize from FP32 down to BF16, which doubles top-k throughput while preserving 99.7% recall.

The trick that makes all of this fit together is a lossless conversion between FP4 and FP8. FP8 has two more exponent bits than FP4, so as long as a per-block scale factor lives within FP8's dynamic range, the quantization is reversible. The whole pipeline runs in the existing FP8 framework.