Concept ladder¶
This is a map of when technical vocabulary enters the canonical course. Every chapter still defines a term inline; this page helps readers see how later ideas depend on earlier ones.
Stage 0 — ordinary ideas¶
First taught in Before the jargon:
| Term | Plain meaning |
|---|---|
| Example or data | Something the program can learn from |
| Pattern | A relationship that appears across examples |
| Model | A program whose behavior was shaped by examples |
| Input / output | What goes into the program / what comes out |
| Prediction | A possible output selected before the answer is known |
| Parameter | An adjustable internal number |
| Training | Adjusting parameters using examples |
| Inference | Using the trained model without changing those parameters |
| Token | A numbered piece of text |
Stage 1 — language-model basics¶
First taught across What an LLM is, Learning from examples, and Text becomes tokens:
| Term | What it adds |
|---|---|
| Context | The earlier text available for the next prediction |
| Vocabulary | The complete set of token IDs a tokenizer can produce |
| Probability | A number describing how likely an option is under the model |
| Loss | One number measuring how poor a training prediction was |
| Optimizer | The rule that turns learning signals into parameter changes |
Stage 2 — the mathematical toolkit¶
First taught in Math with shapes and PyTorch mental model:
| Term | What it adds |
|---|---|
| Vector | An ordered list of numbers |
| Matrix | A rectangular grid of numbers |
| Tensor | A general multi-dimensional collection of numbers |
| Shape | The size of each tensor dimension |
| Gradient | A local signal showing how a small change affects loss |
Stage 3 — the ordinary Transformer¶
First taught in the Transformer section:
- embedding;
- position information;
- query, key, and value;
- causal self-attention;
- attention head;
- feed-forward layer;
- normalization;
- residual connection;
- language-model head and logits.
Stage 4 — training at scale¶
First taught in Pretraining:
- batch and training step;
- learning-rate schedule;
- checkpoint;
- data, tensor, pipeline, context, and sequence parallelism;
- validation and benchmark evaluation.
Stage 5 — optional architecture extensions¶
Only after the dense Transformer is established, the Mixture of experts section introduces:
- dense versus sparse computation;
- expert feed-forward networks;
- router and top-k selection;
- expert capacity and load balancing;
- expert parallelism.
Stage 6 — behavior after pretraining¶
The Post-training section introduces supervised instruction tuning, preference data, reward models, preference optimization, reinforcement learning, verifiers, and reasoning-oriented training.
Stage 7 — running and operating the model¶
The Inference section introduces logits, decoding, temperature, top-k/top-p sampling, key-value cache, batching, quantization, and speculative decoding in their production context.
Stage 8 — systems around the model¶
The Prompting and Agents sections introduce structured output, tool calls, retrieval-augmented generation, memory, planning, authorization, and agent evaluation.