GRPO - Group Relative Policy Optimization - How DeepSeek trains reasoning models

3-Second Blueprint (English)

1. Group Relative Policy Optimization (GRPO) trains reasoning LLMs via RL without a separate critic model by sampling response groups.
2. Step and outcome rewards are normalized across the sampled group to compute relative advantages that weight token probability ratios.
3. PPO-style probability clipping and a KL-divergence penalty constrain policy updates, preventing catastrophic model drift.