Attention is
all you need.

In 2017, the dominant sequence-to-sequence models were recurrent. LSTMs and GRUs read tokens one at a time, threading a hidden state forward through the sequence. The architecture was elegant and powerful and slow. Every step depended on the step before. That dependency made batching across the time axis impossible and made long sequences expensive in two compounding ways: more sequential operations, and a longer path between distant tokens that had to learn to talk to each other.

Convolutional alternatives (ByteNet, ConvS2S) sidestepped the sequential problem by computing positions in parallel, but the receptive field still grew slowly with depth. Distant tokens either took many layers to interact, or required dilations and tricks. Path length between far positions remained O(log n) at best.

The 2017 paper proposed a third option. Drop recurrence. Drop convolution. Connect every position to every other position in one layer through attention, and let the model decide which connections matter. Path length collapses to one. Sequential operations collapse to one. Computational complexity moves from O(n · d²) for a recurrent layer to O(n² · d) for self-attention, which is a win whenever n < d. For typical sentences encoded with d = 512, that is essentially always.

The attention operation takes three inputs. A matrix of queries Q, a matrix of keys K, and a matrix of values V. For every row in Q (one query per token), we compute a dot product against every row in K (one key per token). The result is a matrix of compatibility scores: how much should each query attend to each key.

The scores divide by √dk for a practical reason. Without that scaling, large dk pushes the softmax into a regime where its gradients vanish. The paper hypothesizes that for components of q and k with mean zero and variance one, their dot product has variance dk, large enough to saturate softmax at frontier dimension. Dividing by √dk restores stable gradients.

A softmax over each row turns the scaled scores into attention weights that sum to one. Those weights then weight the rows of V to produce the output: each token's representation is now a content-weighted blend of every other token's value. The whole thing is one matrix multiplication, one scale, one softmax, one multiplication.

A single attention layer has a problem. It averages information from every position the query attends to. Averaging is fine when every relevant relationship lives in the same representation subspace. It is destructive when different relationships need to be tracked at the same time: subject-verb agreement and preposition binding and anaphora and tense, all braided into a single weighted sum.

The fix is to run the attention operation h times in parallel, each in its own learned linear projection of Q, K, and V. Each head sees a different subspace. Each head learns a different attention pattern. The outputs concatenate, then a final linear projection collapses them back to dmodel. Because each head operates on dk = dv = dmodel / h, the total compute stays comparable to single-head with full width.

In the base model, h = 8. Eight parallel attention patterns, eight parallel learned projections, one concatenated output. The paper's appendix shows individual heads learning to perform syntactically and semantically distinct jobs without ever being told to. Some attend to the previous token. Some link verbs to their objects. Some specialize in long-range coreference.

The full Transformer is two stacks of six identical layers each. The encoder reads the input. The decoder writes the output, one token at a time, conditioned on what it has written and on the encoder's output. Every sub-layer is wrapped in a residual connection followed by layer normalization, so the running representation is the sub-layer's output added to its input. The whole stack outputs vectors of dimension dmodel = 512 from start to finish.

Three flavors of attention show up. The encoder uses self-attention over its input. The decoder uses masked self-attention, which prevents each position from looking at future positions during training, preserving the auto-regressive property. The decoder also uses encoder-decoder attention, where queries come from the decoder and keys and values come from the encoder's output. That third flavor is how the decoder grounds its generation in the input sentence.

Position information is added at the bottom of both stacks via sinusoidal positional encodings. Sine and cosine waves of different frequencies, summed onto the embeddings before any attention happens. The paper found that learned positional embeddings worked equally well, but they preferred the sinusoids because they generalize to sequences longer than anything seen during training.

The complexity argument was the most under-appreciated part of the paper at publication. Self-attention costs more per layer in raw FLOPs at long sequence lengths, but it costs much less in sequential operations because every position can be computed at the same time. The maximum path length between any two positions also drops to a constant. The slider below makes that tradeoff legible. Drag n across the typical range of inputs and watch which architecture is cheapest at each length, holding d = 512 constant.

The final ingredient was the result. The Transformer beat every previously reported model, including ensembles, on WMT 2014 English-to-German by more than two BLEU. It set a new single-model state of the art on English-to-French, training in three and a half days on eight P100 GPUs, a small fraction of the cost of the previous best. The base model trained for twelve hours and already beat every competitor.