Improve Benchmarking by JohnReedV · Pull Request #2798 · opentensor/subtensor

JohnReedV · 2026-06-25T16:23:21Z

Description

This PR

Modifies the benchmark flow so that only extrinsics that have drifted passed the threshold or have incorrect read/writes are updated rather than replacing the entire file and creating conflicts everywhere.
Adds a lint to check for extrinsics with missing benchmarks. This can be hidden with
#[allow(unknown_lints, benchmarked_weight_not_plugged)]
Adds 48 missing benchmarks
Fixes a bug where benchmarks could fail on noisy internal component drift even when the actual benchmark-level weight was still within the configured threshold.
Adds block_step to the benchmark flow so the on_initialize weight is generated instead of hardcoded. The benchmark simulates the current worst case with 2 epochs in a block.
Updates the drift check to include proof size and to compare the actual benchmark-level weight instead of failing on noisy internal component drift.

github-actions · 2026-06-25T16:25:44Z

🛡️ AI Review — Skeptic (security review)

VERDICT: VULNERABLE

Baseline scrutiny: established OpenTensor-associated contributor with repo write permission and substantial subtensor history; branch apply-benchmark-to-only-bad-benches -> devnet-ready.

The PR does not modify .github/ai-review/* or .github/copilot-instructions.md. Manifest edits only enable features on existing workspace dependencies; no new external dependency or Cargo.lock change is introduced.

Findings

Sev	File	Finding
HIGH	`pallets/subtensor/src/benchmarks.rs:3080`	Benchmark the full-subnet registration path	inline

Prior-comment reconciliation

2caec2f2: not addressed — The benchmark still sets MaxAllowedUids to 4096 but does not populate the subnet to that size before calling register_limit, so it continues to measure append instead of prune/replace.

Conclusion

The PR appears legitimate, but the prior register_limit underweighting issue remains. Production registration can take the full-subnet prune/replace path while this benchmark still measures an empty-subnet append path.

📜 Previous run (superseded)

~~Sev~~	~~File~~	~~Finding~~	Status
~~HIGH~~	~~`pallets/subtensor/src/benchmarks.rs:3080`~~	~~Benchmark the full-subnet registration path~~	➡️ Carried forward to current findings The benchmark still sets `MaxAllowedUids` to 4096 but does not populate the subnet to that size before calling `register_limit`, so it continues to measure append instead of prune/replace.

🔍 AI Review — Auditor (domain review)

VERDICT: 👎

Gittensor UNKNOWN; author has repo write permission, @opentensor profile context, and substantial recent subtensor history, so calibrated as established-contributor domain review.

PR body is substantive and matches the benchmark/tooling scope, so I would leave it unchanged.

Validation performed: bash -n scripts/benchmark_action.sh passed, and git diff --check origin/devnet-ready...HEAD passed. I did not run cargo tests because the blocking issue is visible from the benchmark setup and dispatch path.

Findings

Sev	File	Finding
HIGH	`pallets/subtensor/src/benchmarks.rs:3080`	Benchmark the full-subnet registration path	inline

Prior-comment reconciliation

21660033: addressed — No conflict markers remain in pallets/proxy/src/weights.rs, and git diff --check origin/devnet-ready...HEAD passes.
ece3dc4d: addressed — No conflict markers remain in pallets/utility/src/weights.rs, and git diff --check origin/devnet-ready...HEAD passes.
2cfa55a3: addressed — The WeightInfo matcher still requires a method-name boundary and retains regression coverage for prefix false positives such as swap_coldkey vs swap_coldkey_announced.

Conclusion

Blocking on the register_limit benchmark: the PR plugs this dispatch into generated weights, but the benchmark measures only the cheap append path while production can take the full-subnet replacement path. That can leave the generated weight materially below the worst-case execution cost.

github-actions · 2026-06-25T16:28:28Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-25T17:53:38Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-25T18:52:27Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-26T16:47:33Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-26T17:17:22Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

…tensor/subtensor into apply-benchmark-to-only-bad-benches

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-06-30T15:05:53Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👎

github-actions · 2026-06-30T15:11:38Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-06-30T17:06:32Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-06-30T18:18:42Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👎

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-06-30T20:53:48Z

🔄 AI review updated — Skeptic: VULNERABLE

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-07-01T13:39:38Z

🔄 AI review updated — Skeptic: VULNERABLE

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-07-01T16:12:02Z

+        let hotkey: T::AccountId = account("register_limit_hot", 0, 1);
+
+        Subtensor::<T>::init_new_network(netuid, 1);
+        Subtensor::<T>::set_max_allowed_uids(netuid, 4096);


[HIGH] Benchmark the full-subnet registration path

register_limit now supplies the production dispatch weight, but this benchmark only initializes an empty subnet, sets MaxAllowedUids to 4096, and then registers one neuron. That measures the cheap current_subnetwork_n < max_allowed append path. In production, once the subnet is full, do_register_limit -> do_register -> register_neuron takes the prune/replace path, including replace_neuron and clear_neuron, which clears existing weights/bonds and iterates stored weight rows. A signed caller can hit that path, so the generated weight can undercharge the expensive registration case. Populate the subnet to MaxAllowedUids with a prune candidate before the extrinsic call, or otherwise benchmark the worst-case full-subnet replacement path.

github-actions · 2026-07-01T16:12:16Z

🔄 AI review updated — Skeptic: VULNERABLE

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-07-01T22:59:23Z

+        let hotkey: T::AccountId = account("register_limit_hot", 0, 1);
+
+        Subtensor::<T>::init_new_network(netuid, 1);
+        Subtensor::<T>::set_max_allowed_uids(netuid, 4096);


[HIGH] Benchmark the full-subnet registration path

This still sets MaxAllowedUids to 4096 but does not populate the subnet before measuring register_limit. In production, do_register_limit delegates into do_register, and register_neuron only appends while SubnetworkN < MaxAllowedUids; once full, it calls get_neuron_to_prune and replace_neuron, which clears/rebuilds substantially more storage. The benchmark therefore keeps producing an append-path weight for an extrinsic that users can invoke on the heavier full-subnet prune/replace path. Seed the subnet to MaxAllowedUids with a pruneable candidate before the measured call, or benchmark the worst reachable path explicitly.

github-actions · 2026-07-01T22:59:33Z

🔄 AI review updated — Skeptic: VULNERABLE

Update benchmark_action.sh

29708f2

JohnReedV added the skip-cargo-audit This PR fails cargo audit but needs to be merged anyway label Jun 25, 2026

JohnReedV added the apply-benchmark-patch label Jun 25, 2026

auto-update benchmark weights

4239fd8

github-actions Bot removed the apply-benchmark-patch label Jun 25, 2026

JohnReedV added 2 commits June 25, 2026 10:29

bump spec

900fbd4

fix bad failures

a80f02e

JohnReedV added the apply-benchmark-patch label Jun 25, 2026

auto-update benchmark weights

538098e

github-actions Bot removed the apply-benchmark-patch label Jun 25, 2026

test benchmark

f48cb35

JohnReedV added the apply-benchmark-patch label Jun 25, 2026

auto-update benchmark weights

524e551

github-actions Bot removed the apply-benchmark-patch label Jun 25, 2026

JohnReedV added the apply-benchmark-patch label Jun 26, 2026

auto-update benchmark weights

46bff83

github-actions Bot removed the apply-benchmark-patch label Jun 26, 2026

JohnReedV added 2 commits June 26, 2026 09:23

add block_step benchmark

36ae2e5

add benchmark lint

37d9bed

JohnReedV and others added 2 commits June 26, 2026 10:11

fix lint

7d97e23

chore: auditor auto-fix

38109a8

JohnReedV changed the title ~~Apply Benchmark patch only where its needed~~ Improve Benchmarking Jun 26, 2026

JohnReedV added 2 commits June 26, 2026 10:40

lint ignores mock benches

af5d43b

Merge branch 'apply-benchmark-to-only-bad-benches' of github.com:open…

952a23a

…tensor/subtensor into apply-benchmark-to-only-bad-benches

JohnReedV added 2 commits June 30, 2026 08:00

revert

e14d5df

fix merge

a0de6ba

github-actions Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread pallets/proxy/src/weights.rs Outdated

Comment thread pallets/utility/src/weights.rs Outdated

fix lint tests

88b970e

JohnReedV removed the apply-benchmark-patch label Jun 30, 2026

stabilize some

8388cd7

github-actions Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread pallets/subtensor/src/benchmarks.rs

Update benchmarking.rs

c2639fb

github-actions Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread pallets/subtensor/src/benchmarks.rs

JohnReedV added 2 commits July 1, 2026 06:36

fix limit order benchmark

62648d1

improve logs

b469aa4

github-actions Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread pallets/subtensor/src/benchmarks.rs

JohnReedV added the apply-benchmark-patch label Jul 1, 2026

auto-update benchmark weights

2cf9bc9

github-actions Bot removed the apply-benchmark-patch label Jul 1, 2026

improve logs

93d0dd6

github-actions Bot reviewed Jul 1, 2026

View reviewed changes

JohnReedV added the apply-benchmark-patch label Jul 1, 2026

auto-update benchmark weights

511be8c

github-actions Bot removed the apply-benchmark-patch label Jul 1, 2026

fix compile

bebf56c

github-actions Bot reviewed Jul 1, 2026

View reviewed changes

Uh oh!

Conversation

JohnReedV commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

github-actions Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛡️ AI Review — Skeptic (security review)

Findings

Prior-comment reconciliation

Conclusion

🔍 AI Review — Auditor (domain review)

Findings

Prior-comment reconciliation

Conclusion

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JohnReedV commented Jun 25, 2026 •

edited

Loading

github-actions Bot commented Jun 25, 2026 •

edited

Loading