TL;DR
The Transformer, proposed in Google's 2017 paper "Attention Is All You Need," is built on self-attention: each token computes attention weights over every other token using Query/Key/Value vectors, modeling arbitrary-distance dependencies directly. With no recurrent sequential dependency, training parallelizes fully, making it the foundation of GPT, BERT, and most modern LLMs.
Core components
| Component | Role |
|---|---|
| Self-attention | Every token attends to all tokens; long-range dependencies |
| Multi-head attention | Parallel QKV heads capture different subspaces |
| Positional encoding | Injects order (attention is order-agnostic) |
| Feed-forward network | Per-position nonlinear transformation |
| Residuals + LayerNorm | Stable deep training |
Attention math (QKV)
- Multiply inputs by weight matrices to get Q, K, V;
- Compute Q·Kᵀ and scale by √d_k;
- Apply softmax to get attention weights;
- Multiply weights by V to get the output.
Two dominant variants
- Encoder-only (BERT): understanding tasks (classification, retrieval);
- Decoder-only (GPT): generation tasks; masked self-attention sees only already-generated tokens.
FAQ
Why is the Transformer faster than RNNs?
RNNs compute step by step and cannot parallelize; Transformers process the whole sequence at once, making training orders of magnitude faster.
What is positional encoding for?
Attention alone is order-agnostic (shuffling inputs gives the same result); positional encoding injects sequence order so the model knows what comes before what.
Do GPT and BERT both use Transformers?
Yes. GPT uses the decoder (decoder-only, autoregressive generation); BERT uses the encoder (encoder-only, bidirectional understanding). Both rely on multi-head self-attention.
Sources
- Vaswani et al., Attention Is All You Need, arXiv:1706.03762 (2017), accessed 2026-08-04
- Jay Alammar, The Illustrated Transformer, accessed 2026-08-04