TL;DR
The Transformer (Vaswani et al., 2017, "Attention Is All You Need") replaces recurrence with self-attention, enabling parallelism and long-range modeling. It is the foundation of modern large language models.
Key Points
- Self-attention: each token produces Query/Key/Value vectors; outputs are weighted sums based on Q·K similarity.
- Multi-head attention: several attention heads in parallel capture different relationship subspaces.
- Position encoding: attention is order-agnostic, so position must be injected (sinusoidal, learned, or RoPE).
- Block structure: attention + feed-forward + residual connections + layer norm.
- Two forms: encoder-decoder (T5) and decoder-only (GPT-style); LLMs are predominantly decoder-only autoregressive models.
- Why fast: tokens can be processed in parallel (unlike RNNs); the cost is O(n²) attention complexity.
Further Notes
- Scaling laws: capability improves predictably with model size, data, and compute.
- KV cache: caching past Key/Value pairs speeds up inference.
- Sparse/linear attention reduce long-context cost.
Sources
- Vaswani et al., "Attention Is All You Need" (2017)
- Wikipedia: Transformer (deep learning architecture)
- Model technical reports (GPT, Llama, DeepSeek)