★ Attention Is All You Need

Summary

TL;DR. The Transformer replaces recurrence and convolution with attention alone, making sequence transduction fully parallelizable and setting new SOTA on WMT’14 EN-DE (28.4 BLEU) and EN-FR (41.8 BLEU) at a fraction of prior training cost.

Key contributions

• An encoder–decoder architecture built entirely on attention — no RNNs or convolutions — so computation parallelizes across sequence positions.
• Scaled dot-product attention, scaling scores by $1/\sqrt{d_k}$ to keep softmax gradients usable at large key dimensions.
• Multi-head attention ($h = 8$) to attend jointly to different representation subspaces, at roughly the cost of single-head attention.
• Sinusoidal positional encodings to inject order without recurrence.
• New SOTA on two translation tasks plus strong English constituency parsing, trained in as little as 12 hours on 8 P100 GPUs (base model).

Detailed Explanation

The problem. RNN/LSTM sequence models factor computation along token positions, generating hidden state $h_t$ from $h_{t-1}$. This inherently sequential dependency blocks parallelization within a training example and makes batching memory-bound at long sequence lengths.

Prior solutions and their gaps. Attention had been added on top of recurrence to model long-range dependencies regardless of distance. Convolutional models (ByteNet, ConvS2S) parallelize but the operations needed to relate two positions grow with their distance — linearly (ConvS2S) or logarithmically (ByteNet) — making distant dependencies harder to learn. None dropped recurrence entirely.

How this closes the gap. Self-attention relates any two positions in a constant number of sequential operations (Table 1: $O(1)$ sequential ops and $O(1)$ maximum path length, vs. $O(n)$ for recurrence). Shorter paths between positions make long-range dependencies easier to learn, while every position is computed in parallel.

Novel ideas. Scaled dot-product attention,

where the $1/\sqrt{d_k}$ factor counteracts the large-magnitude dot products that otherwise push softmax into low-gradient regions. Multi-head attention projects $Q$/$K$/$V$ into $h = 8$ subspaces ($d_k = d_v = 64$) so heads specialize; the authors note single-head averaging would inhibit this. Positional encodings use sinusoids of geometrically spaced wavelengths, chosen partly so the model might extrapolate to longer sequences than seen in training.

Method. Both encoder and decoder are stacks of $N = 6$ identical layers, $d_{\text{model}} = 512$. Each encoder layer = multi-head self-attention + a position-wise feed-forward net ($d_{\text{ff}} = 2048$, ReLU), each wrapped in a residual connection and layer norm. The decoder adds a third sub-layer attending over encoder outputs, and masks self-attention so position $i$ can only attend to positions $< i$, preserving auto-regression. Trained with Adam ($\beta_1 = 0.9$, $\beta_2 = 0.98$), an inverse-square-root LR schedule with 4000 warmup steps, residual dropout 0.1, and label smoothing 0.1 (which hurts perplexity but helps BLEU).

Results. The big model reaches 28.4 BLEU (EN-DE) and 41.8 BLEU (EN-FR), beating prior models and ensembles while using markedly fewer training FLOPs (Table 2). Ablations (Table 3) show single-head attention is 0.9 BLEU worse than the best head count, reducing $d_k$ hurts quality (suggesting compatibility is non-trivial), and learned positional embeddings perform nearly identically to sinusoids. On WSJ constituency parsing the model reaches 91.3 F1 (WSJ-only) and 92.7 (semi-supervised) with little task-specific tuning.

Critique (open questions)

• The headline efficiency claim rests on FLOP estimates derived from training-time × GPU-count × an assumed sustained TFLOPS — how robust is the “fraction of the cost” comparison to that estimation, given the baselines ran on different hardware?
• Self-attention is $O(n^2 \cdot d)$ per layer; the paper notes restricted attention as future work — so what evidence is there that the approach holds up on long sequences, where the quadratic term dominates?
• Sinusoidal encodings are justified by a hypothesis that they “may allow” extrapolation to longer sequences, yet row (E) shows learned embeddings match them — is the extrapolation benefit ever actually demonstrated, or only asserted?
• Results concentrate on two high-resource translation pairs plus one parsing task; what supports the broad “generalizes well to other tasks” framing beyond these, especially for low-resource or non-text modalities?
• Ablations vary one axis at a time on a single dev set (newstest2013) — how much do the chosen interactions (heads × d_k, model size × dropout) hide, and are the differences within run-to-run variance?

My Notes

My own notes go here — this file is never overwritten by the ingest pipeline.

• Re-read the multi-head attention section; sketch the shapes by hand.
• Compare positional encoding choices vs. learned embeddings.