Video course · 15 chapters · 68 min · certificate
DeepSeek R1 Theory Tutorial – Architecture, GRPO, KL Divergence
The theory behind DeepSeek R1: the R1-Zero path, the reinforcement learning setup, GRPO and its advantage, R1-Zero results, cold-start SFT, consistency rewards, neural reward models and distillation, PPO vs GRPO formulas and trainer code, and KL divergence estimators k1, k2 and k3.
What you'll learn
- Explain DeepSeek R1's training pipeline
- Describe GRPO and how it differs from PPO
- Interpret R1-Zero results
- Explain cold-start SFT and distillation
- Read GRPO pseudo-code and trainer code
- Compare KL divergence estimators
Chapters
15 chapters · 67:43-
5:32
01Overview Members
R1 overview and the R1-Zero path
Pure RL.
-
3:04
02RL Members
Reinforcement learning setup
Rewards.
-
4:28
03GRPO Members
GRPO
Group-relative advantage.
-
3:49
04Results Members
R1-Zero results
Close to o1.
-
4:13
05SFT Members
Cold-start SFT and consistency
Fixing readability.
-
5:49
06Distillation Members
Neural reward models and distillation
General data.
-
3:30
07Comparison Members
PPO vs GRPO
From DeepSeekMath.
-
3:00
08PPO Members
PPO formula
Clipped objective.
-
5:31
09GRPO math Members
GRPO formula and pseudo-code
Group averages.
-
10:28
10Code Members
GRPO trainer code
Open source.
-
1:56
11KL Members
KL divergence in GRPO vs PPO
The penalty term.
-
4:12
12KL basics Members
KL divergence refresher
Broad recap.
-
4:29
13Estimation Members
Monte Carlo KL and k1
Schulman's blog.
-
4:43
14Estimators Members
k2 and k3 estimators
Bias and variance.
-
2:59
15Wrap-up Members
Benchmarking and conclusion
Recap.