Repository navigation
Duty scheduler: the last slot of each sync-committee period runs the outgoing committee instead of the incoming one (contributions rejected with AggregatorNotInCommittee, incoming members miss the slot) #3052
Description
Summary
In the last slot of every sync-committee period, the node executes sync-committee duties (messages and contributions) for the outgoing period's committee. Per the Altair validator spec, that slot belongs to the incoming committee: a committee assigned to a slot signs for slot - 1, so a committee assigned to an epoch signs from one slot before the epoch starts (spec, "Sync Committee"). The subnet computation reflects this: compute_subnets_for_sync_committee switches to state.next_sync_committee when compute_epoch_at_slot(state.slot + 1) falls in the next period (spec).
So once per period (256 epochs, ~27h18m):
- Members joining the committee miss that slot. Incoming-only members produce no sync-committee message for the period's last slot, and are never considered for contribution aggregation in it, so they don't take part in the sync aggregate of the new period's first block.
- Messages from members leaving the committee are wasted. They can never be included. Lighthouse returns 200 but publishes nothing (the validator has no subnet position in the next slot's committee), so this part is invisible.
- Contributions from outgoing aggregators are rejected with
AggregatorNotInCommittee. Every operator gets anERRORlog and assv_runner_submissions_failed_total{ssv_beacon_role="SYNC_COMMITTEE_CONTRIBUTION"}increment for each one.
The codebase already notes that a message from the outgoing committee in that final slot would never be included (LastSlotOfSyncPeriod in v2.4.3, LastActionableSlotOfSyncPeriod on stage). But that helper only drives duty fetching and cleanup; it doesn't change which committee executes the slot.
The per-validator cost is small: one slot of sync-committee participation per period, for members joining the committee. But it is systematic, it applies to every network including mainnet, and it produces recurring errors that look like beacon-node problems.
Affected code
Both executors select sync duties by the period of the current slot's epoch.
Sync-committee messages: operator/duties/committee.go (L100). On stage the same CommitteeHandler also runs the post-Boole AGGREGATOR_COMMITTEE flavour, so post-Boole contributions go through this path too.
currentEpoch := h.beaconConfig.EstimatedEpochAtSlot(currentSlot)
currentPeriod := h.beaconConfig.EstimatedSyncCommitteePeriodAtEpoch(currentEpoch)
// ...
h.processExecution(tickCtx, currentPeriod, currentEpoch, currentSlot)
// processExecution:
syncDuties := h.syncDuties.CommitteePeriodDuties(period)Contributions (pre-Boole): operator/duties/sync_committee.go (L229).
currentPeriod := h.beaconConfig.EstimatedSyncCommitteePeriodAtEpoch(currentEpoch)
// ...
h.processExecution(tickCtx, currentPeriod, currentSlot)
// processExecution:
duties := h.duties.CommitteePeriodDuties(period)This is unchanged on stage: CommitteeHandler.processExecution and SyncCommitteeHandler.processExecution still receive currentPeriod, derived from the current slot's epoch.
Evidence: Sepolia, 4-operator cluster (100 validators), v2.4.3, Lighthouse v8.1.3
Every contribution failure lines up with a period's last slot. Over ~57 days of uptime, each operator recorded 155 SYNC_COMMITTEE_CONTRIBUTION submission failures, identically on all four. They came in bursts of 1–9 at almost every period boundary. Every failure whose time can be pinned down falls in a period's last slot; 147 of the 155 can be timed, and the other 8 fall in a metrics gap.
Example: slot 11206655 (epoch 350207, slot 31 of the epoch), the last slot of period 1367. All four operators logged the same thing:
✅ successfully submitted sync committeefor 23 validators in slot 11206655. That is the outgoing committee, the same set as slot 11206654.- From slot 11206656 on, the same log line covers 26 validators, the incoming committee.
- 7 validators sit in both committees, so their messages still counted.
- The 19 incoming-only members produced no message for slot 11206655.
- The 16 outgoing-only messages were accepted with a 200 and dropped.
- Five contributions from outgoing aggregators were rejected:
POST failed with status 400: {"code":400,"message":"BAD_REQUEST: error processing contribution and proofs","failures":[{"index":0,"message":"Verification: AggregatorNotInCommittee { aggregator_index: 1658 }"}]}
The same error came back for aggregators 1598, 1582, 1660 and 1574.
I haven't inspected the new period's first block on-chain. The missed participation follows from the spec and from no message being produced.
Reference behaviour (Lighthouse)
- Validator client: schedules sync duties for
wall_clock_slot + 1(get_duties_for_slot). - Beacon node: computes a sync message's subnets from the next slot's committee (
get_subnet_positions_for_sync_committee_message). It then verifies and publishes only on those subnets. An empty set means nothing is published and no failure is reported, so the call returns 200 (process_sync_committee_signatures).
Suggested fix
Take the sync-committee duty period for execution from the next slot, as the spec and Lighthouse's validator client do:
dutyPeriod := h.beaconConfig.EstimatedSyncCommitteePeriodAtEpoch(
h.beaconConfig.EstimatedEpochAtSlot(currentSlot + 1),
)Use it for CommitteePeriodDuties(...) in CommitteeHandler.processExecution (both flavours) and SyncCommitteeHandler.processExecution. Things to keep in mind:
- Duty availability: the next period's duties must be in the store by the period's last slot. They are pre-fetched
preparationSlotsahead of the boundary, but make sure the fetch intents and theReset(currentPeriod - 1)cleanup keep them available at that slot. - Duty slot: the duty's
Slotstays the current slot. Only membership (andValidatorSyncCommitteeIndices) comes from the next period. - Test: add a boundary unit test.
- The last slot of
PexecutesP+1's members, including their contribution duties. P's outgoing-only members don't execute in that slot.- The first slot of
P+1is unchanged.
- The last slot of
Activity