Skip to content

Duty scheduler: the last slot of each sync-committee period runs the outgoing committee instead of the incoming one (contributions rejected with AggregatorNotInCommittee, incoming members miss the slot) #3052

Open

Description

@iurii-ssv

Summary

In the last slot of every sync-committee period, the node executes sync-committee duties (messages and contributions) for the outgoing period's committee. Per the Altair validator spec, that slot belongs to the incoming committee: a committee assigned to a slot signs for slot - 1, so a committee assigned to an epoch signs from one slot before the epoch starts (spec, "Sync Committee"). The subnet computation reflects this: compute_subnets_for_sync_committee switches to state.next_sync_committee when compute_epoch_at_slot(state.slot + 1) falls in the next period (spec).

So once per period (256 epochs, ~27h18m):

  1. Members joining the committee miss that slot. Incoming-only members produce no sync-committee message for the period's last slot, and are never considered for contribution aggregation in it, so they don't take part in the sync aggregate of the new period's first block.
  2. Messages from members leaving the committee are wasted. They can never be included. Lighthouse returns 200 but publishes nothing (the validator has no subnet position in the next slot's committee), so this part is invisible.
  3. Contributions from outgoing aggregators are rejected with AggregatorNotInCommittee. Every operator gets an ERROR log and a ssv_runner_submissions_failed_total{ssv_beacon_role="SYNC_COMMITTEE_CONTRIBUTION"} increment for each one.

The codebase already notes that a message from the outgoing committee in that final slot would never be included (LastSlotOfSyncPeriod in v2.4.3, LastActionableSlotOfSyncPeriod on stage). But that helper only drives duty fetching and cleanup; it doesn't change which committee executes the slot.

The per-validator cost is small: one slot of sync-committee participation per period, for members joining the committee. But it is systematic, it applies to every network including mainnet, and it produces recurring errors that look like beacon-node problems.


Affected code

Both executors select sync duties by the period of the current slot's epoch.

Sync-committee messages: operator/duties/committee.go (L100). On stage the same CommitteeHandler also runs the post-Boole AGGREGATOR_COMMITTEE flavour, so post-Boole contributions go through this path too.

currentEpoch := h.beaconConfig.EstimatedEpochAtSlot(currentSlot)
currentPeriod := h.beaconConfig.EstimatedSyncCommitteePeriodAtEpoch(currentEpoch)
// ...
h.processExecution(tickCtx, currentPeriod, currentEpoch, currentSlot)

// processExecution:
syncDuties := h.syncDuties.CommitteePeriodDuties(period)

Contributions (pre-Boole): operator/duties/sync_committee.go (L229).

currentPeriod := h.beaconConfig.EstimatedSyncCommitteePeriodAtEpoch(currentEpoch)
// ...
h.processExecution(tickCtx, currentPeriod, currentSlot)

// processExecution:
duties := h.duties.CommitteePeriodDuties(period)

This is unchanged on stage: CommitteeHandler.processExecution and SyncCommitteeHandler.processExecution still receive currentPeriod, derived from the current slot's epoch.


Evidence: Sepolia, 4-operator cluster (100 validators), v2.4.3, Lighthouse v8.1.3

Every contribution failure lines up with a period's last slot. Over ~57 days of uptime, each operator recorded 155 SYNC_COMMITTEE_CONTRIBUTION submission failures, identically on all four. They came in bursts of 1–9 at almost every period boundary. Every failure whose time can be pinned down falls in a period's last slot; 147 of the 155 can be timed, and the other 8 fall in a metrics gap.

Example: slot 11206655 (epoch 350207, slot 31 of the epoch), the last slot of period 1367. All four operators logged the same thing:

  • ✅ successfully submitted sync committee for 23 validators in slot 11206655. That is the outgoing committee, the same set as slot 11206654.
  • From slot 11206656 on, the same log line covers 26 validators, the incoming committee.
  • 7 validators sit in both committees, so their messages still counted.
  • The 19 incoming-only members produced no message for slot 11206655.
  • The 16 outgoing-only messages were accepted with a 200 and dropped.
  • Five contributions from outgoing aggregators were rejected:
POST failed with status 400: {"code":400,"message":"BAD_REQUEST: error processing contribution and proofs","failures":[{"index":0,"message":"Verification: AggregatorNotInCommittee { aggregator_index: 1658 }"}]}

The same error came back for aggregators 1598, 1582, 1660 and 1574.

I haven't inspected the new period's first block on-chain. The missed participation follows from the spec and from no message being produced.


Reference behaviour (Lighthouse)


Suggested fix

Take the sync-committee duty period for execution from the next slot, as the spec and Lighthouse's validator client do:

dutyPeriod := h.beaconConfig.EstimatedSyncCommitteePeriodAtEpoch(
	h.beaconConfig.EstimatedEpochAtSlot(currentSlot + 1),
)

Use it for CommitteePeriodDuties(...) in CommitteeHandler.processExecution (both flavours) and SyncCommitteeHandler.processExecution. Things to keep in mind:

  • Duty availability: the next period's duties must be in the store by the period's last slot. They are pre-fetched preparationSlots ahead of the boundary, but make sure the fetch intents and the Reset(currentPeriod - 1) cleanup keep them available at that slot.
  • Duty slot: the duty's Slot stays the current slot. Only membership (and ValidatorSyncCommitteeIndices) comes from the next period.
  • Test: add a boundary unit test.
    • The last slot of P executes P+1's members, including their contribution duties.
    • P's outgoing-only members don't execute in that slot.
    • The first slot of P+1 is unchanged.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions