About global batch size for reproducing paper results

[GoldMilkyway]

Hi, thank you for the impressive work and for releasing the code.

I'm trying to reproduce the GDN-2 results from the paper, but I'm confused about the global batch size.

The paper/README state that the 1.3B models were trained with a global batch size of '0.5M' tokens and sequence length '4K'. My understanding is that this corresponds to 128 x 4K sequences.

However, in pretrain.py, I only see branches for 512x4k, 1024x4k, 256x8k, 128x16k, 64x32k, and 1024x2k. I couldn't find a 128x4k configuration. (Or did you use 4 nodes and compute a global batch size only for a node?)

Am I misunderstanding the effective batch-size calculation, or was another config used for the reported experiments?

Thanks!

[ngocbh]

I'm also trying to reproduce the result. I retrained the recurrent-only Gated DeltaNet-2
1.3B (gdn2_1.3B, all layers GDN-2, no attention) on 100B FineWeb-Edu tokens,
following the recipe in your README exactly, and I'm seeing a large gap on the
long-context retrieval (RULER) numbers that I can't explain. Perplexity and
commonsense land right on your reported values, so training itself looks healthy.

Setup (matches README §Training recipe)

• Model: gdn2_1.3B, pure recurrent (gdn2_per_layer=1 → every layer is GatedDeltaNet2, zero attention layers)
• AdamW, peak LR 4e-4, weight decay 0.1, grad clip 1.0
• Cosine schedule, 1B-token warmup, cosine floor min_lr = 4e-5 (peak/10)
• Global batch 0.5M tokens (128 × 4K), sequence length 4K, 100B tokens total
• Data: FineWeb-Edu 100BT sample
• Eval: lm-evaluation-harness 0.4.12, RULER tasks with max_seq_lengths = [4096] (S-NIAH-2, MK-NIAH-1) and [2048] (S-NIAH-3)

Results (our recurrent checkpoint vs. your README "Recurrent" row)

Language modeling / commonsense — on target:

Metric	Ours	README (GDN-2, recurrent)
wikitext ppl ↓	16.27	15.90
lambada ppl ↓	10.96	11.41
lambada acc ↑	49.33	48.09

RULER / NIAH — large regression:

Task	Ours	README (GDN-2, recurrent)
S-NIAH-2 @4k	61.0	93.0
S-NIAH-3 @2k	70.0	89.8
MK-NIAH-1 @4k	30.0	37.8

Perplexity/commonsense reproduce within noise, but recurrent NIAH is ~20–30
points low. Since LM metrics match, this looks like either a training detail
not captured in the README recipe, or an eval-config difference on the RULER
tasks.

Questions

1. LR decay floor: does the cosine schedule decay to 0 or to 0.1×peak
   (4e-5)? We used 4e-5. Could the nonzero floor materially hurt recurrent recall?
2. RULER eval config for the recurrent numbers: which lm-eval-harness
   version, and what exact RULER settings (context assembly, max_seq_lengths,
   tokenizer, any generate_until/greedy vs. loglikelihood scoring) produced
   the 93.0 / 89.8 / 37.8 recurrent row?
3. Checkpoint used for the table: final end-of-cosine checkpoint, best, or an EMA/averaged one?
4. Any recurrent-only inference detail (chunk size, state init, dtype during
   eval) that affects retrieval but not perplexity?
5. Data: is FineWeb-Edu 100BT sample the exact corpus, or a specific
   shuffle/subset/mix used for the reported numbers?

Happy to share full configs/logs. Thanks!