Paper trail¶
This is a route through original papers, not a completeness contest. Read the question, method, assumptions, evidence, and limitations before inheriting the headline.
Representation and architecture¶
| Work | Read it for |
|---|---|
| Neural Machine Translation of Rare Words with Subword Units | neural BPE motivation and merge procedure |
| SentencePiece | raw-text subword training and language-independent tooling |
| Attention Is All You Need | scaled dot-product attention and the original Transformer |
| RoFormer | rotary position embeddings |
| GLU Variants Improve Transformer | gated feed-forward variants including SwiGLU experiments |
| GQA | grouped-query attention and quality/speed trade-offs |
Data and scaling¶
| Work | Read it for |
|---|---|
| The Pile | diverse open corpus design and datasheet |
| Dolma | open corpus construction, tooling, and ablations |
| FineWeb | large-scale web-data filtering and evaluation |
| Scaling Laws for Neural Language Models | empirical loss scaling under its studied regime |
| Training Compute-Optimal Large Language Models | compute allocation between parameters and tokens |
Scaling laws are fitted observations under specific data, architecture, and optimization choices—not physical constants.
Mixture of experts¶
| Work | Read it for |
|---|---|
| Sparsely-Gated Mixture-of-Experts | conditional computation, noisy gating, and balancing |
| GShard | scaling conditional computation with automatic sharding |
| Switch Transformers | top-1 routing, capacity, auxiliary loss, and scale |
| ST-MoE | stable training and transfer behavior |
| Mixtral of Experts | sparse decoder model report and per-token expert activation |
| DeepSeekMoE | fine-grained and shared expert design |
| DeepSeek-V3 | published architecture and auxiliary-loss-free routing approach |
| OLMoE | open MoE training artifacts and analysis |
Post-training and reasoning¶
| Work | Read it for |
|---|---|
| InstructGPT | demonstration data, preference data, reward modeling, and PPO pipeline |
| Constitutional AI | model-generated critique/revision and AI feedback framework |
| Direct Preference Optimization | preference optimization without an explicit learned reward model loop |
| Self-Instruct | synthetic instruction generation and filtering |
| Let's Verify Step by Step | process versus outcome supervision experiments for math |
| DeepSeek-R1 | published RL-centered reasoning model development |
Inference¶
| Work | Read it for |
|---|---|
| FlashAttention | exact attention with IO-aware tiling |
| FlashAttention-2 | improved work partitioning and parallelism |
| vLLM / PagedAttention | paged KV-cache memory management and serving throughput |
| Fast Inference from Transformers via Speculative Decoding | exact-distribution draft/verify decoding |
| SmoothQuant | post-training activation/weight quantization |
Prompts, retrieval, and agents¶
| Work | Read it for |
|---|---|
| Retrieval-Augmented Generation | coupling learned generation with retrieved evidence |
| Chain-of-Thought Prompting | demonstrations with intermediate reasoning on studied models/tasks |
| Self-Consistency | sampling multiple reasoning paths and aggregating answers |
| ReAct | interleaving reasoning-like traces and external actions |
| Toolformer | self-supervised API-use training |
These works report empirical results, not universal prompt laws. Re-test on the exact model, task, tools, and evaluation protocol you operate.