When I set beta=0.0, why does the loss of GRPO always remain 0?

[ctjian]

[St4r4x]

@body_trl_3123.txt

[Mahesh-Paul]

In GRPO (Group Relative Policy Optimization), the loss function consists of two primary components:

1. The policy objective weighted by group-relative advantages.
2. The KL divergence penalty scaled by the coefficient beta ($\beta \cdot D_{KL}$).

When you set beta = 0.0:

• The KL divergence penalty term is multiplied by 0 and completely dropped.
• Additionally, if your reward values across all sampled completions within the group are identical (or normalized advantages result in zero variance), the relative advantage term collapses to zero.
• Without the KL regularization term to produce non-zero reference loss, the total computed loss evaluates to 0.0.

To fix/verify this:

• Ensure your reward function provides non-zero variance across completions in the group.
• Keep a small non-zero beta value (e.g., beta=0.01 or beta=0.04) if you want the KL divergence penalty active to stabilize updates and log non-zero loss values.