Source-code map¶
Use this page as a guided index. Open one teaching implementation and one production implementation for the same mechanism. Default-branch links are for orientation; pin a release or commit for reproducible analysis.
Model forward pass¶
| Question | Compact trail | Scaled trail |
|---|---|---|
| How does a decoder block fit together? | nanoGPT model.py | torchtitan Llama model |
| How are multiple architectures configured? | LitGPT config.py | Hugging Face model implementations |
| How does this book's model work? | src/open_llm_lab/model.py | compare with the two trails above |
Read in this order: config → embedding → one block → attention → FFN → final norm → language-model head → loss.
Attention and position¶
| Mechanism | Source trail |
|---|---|
| Explicit causal attention | companion attention.py |
| Production Transformer attention abstraction | Megatron-Core attention.py |
| FlashAttention kernels and interface | official FlashAttention repository |
| Llama attention/rotary path | torchtitan Llama model |
Ask whether the code is expressing the mathematical layer, selecting a kernel, managing a cache, or distributing tensors. Those concerns can make equivalent attention look unrelated.
Mixture of experts¶
| Mechanism | Source trail |
|---|---|
| Teaching top-k route/combine | companion moe.py |
| Router | Megatron-Core router.py |
| MoE layer orchestration | Megatron-Core moe_layer.py |
| Token dispatch and all-to-all | Megatron-Core token_dispatcher.py |
| Shared experts | Megatron-Core shared_experts.py |
| Open research model | OLMoE repository trail |
Trace a token index, not only a vector: router logits → selected expert IDs → permutation/dispatch → expert batch → weighted combine → inverse permutation → residual stream.
Training systems¶
| Layer | Official codebase | Entry point |
|---|---|---|
| Small GPT training | nanoGPT | train.py |
| Clean multi-model training | LitGPT | project pretrain/finetune commands and litgpt package |
| PyTorch-native distributed LLM training | torchtitan | torchtitan/train.py and parallel dimensions |
| Large-scale Transformer/MoE training | Megatron-LM | megatron/training/training.py |
| Reproducible OLMo runs | OLMo-core | src/scripts/official/ and model cards |
Data¶
| Task | Official codebase |
|---|---|
| OLMo/Dolma processing | Dolma toolkit |
| large-scale text processing | DataTrove |
| RedPajama processing | RedPajama-Data |
| Common Crawl access | Common Crawl examples and index docs |
| tokenizer implementation | SentencePiece and Hugging Face Tokenizers |
Post-training¶
| Task | Official codebase |
|---|---|
| SFT, DPO, reward modeling | Hugging Face TRL |
| open instruction and RLVR recipes | Ai2 Open Instruct |
| distributed RLHF | OpenRLHF |
| large-scale RL | verl |
Objective names are not enough to reproduce a run. Follow data formatting, chat template, masking, reference-model behavior, rollout generation, reward normalization, and distributed launch configuration.
Inference¶
| Question | Official source |
|---|---|
| How are requests scheduled continuously? | vLLM scheduler |
| How are KV blocks managed? | vLLM KV-cache manager |
| How does a serving engine combine kernels and radix/prefix reuse? | SGLang |
| How can quantized inference run broadly? | llama.cpp |
| How does GPU-optimized deployment work? | TensorRT-LLM |
Evaluation¶
Read the prompt templates, answer extraction, few-shot selection, normalization, and task version before comparing a score.