Skip to content

Avoid NaN gradients for balanced routing losses - #218

Merged
jpuigcerver merged 1 commit into
google-research:mainfrom
sylvesterkaczmarek:fix-balanced-routing-gradients
Sep 11, 2026
Merged

jpuigcerver merged 1 commit into
google-research:mainfrom
sylvesterkaczmarek:fix-balanced-routing-gradients

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor

The squared coefficient-of-variation losses differentiate through std, producing NaN gradients at perfectly balanced expert loads. This also affects a zero-initialized router despite its finite loss.

Compute the variance directly in the importance and load losses. Cover balanced gates, one-expert routing and router parameter gradients in eager and JIT execution.

Validation: 76 routing, ensemble-routing and MoE tests pass on four virtual CPU devices (two device-count skips). All 22 new regression cases fail on unchanged upstream. Sixty non-balanced value/gradient comparisons match upstream. Syntax and git diff --check pass.

@jpuigcerver
jpuigcerver merged commit 6dcc8c0 into google-research:main Sep 11, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants