Repository navigation
docs: review perf methodology and plan replacement harness - #526
Merged
Merged
Conversation
Document the methodology problems in tests/perf.mjs, define the key performance metrics and how to measure them, and lay out a phased plan for a replacement benchmark runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Take the memory baseline before fixture setup, move the bounded-memory check to a heap-capped script outside npm test, bootstrap at the process level, align Phase 1 scope with the review, keep CLAUDE.md in step with the phases, and link review line references to a fixed commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bound Buffer memory as well as the V8 heap, stream large generated inputs from a separate process, run 10 processes for the process-level bootstrap, remove memory mode in Phase 1, and keep the CLAUDE.md note warning against before/after comparisons until Phase 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the A/A noise-floor criterion to Phase 4, write CI comparisons to the job summary, keep the bench:compare pointer in Phase 6, generate bare URLs in-process, and add split-and-index to the bounded-memory check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use hyperfine (2.x JSON) to run each scenario in fresh processes and record wall time, CPU time and peak RSS, and Bencher to track history, apply t-test thresholds and report on pull requests, including the workflow_run pattern for forks. Custom code shrinks to scenario scripts, a BMF converter, local A/B comparison and the bounded-memory check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/performance/methodology-review.md, which lists 17 problems with howtests/perf.mjsmeasures performance. Four of them produce wrong numbers: passingfalseturns on memory mode, an unknown test name silently runsstream, memory mode reports the process's lifetime peak RSS, and failed runs still count as samples.docs/performance/measurement-plan.md, which defines the key metrics (throughput, memory growth with input size, peak memory, CPU time per item, validation overhead, startup time) and how to measure each one. The plan builds on hyperfine 2.x to run each scenario in fresh processes and Bencher to track history, flag regressions and report on pull requests, including PRs from forks. Custom code is limited to scenario scripts, a hyperfine-to-Bencher converter, a local before-and-after comparison, and a heap-capped bounded-memory check. It lays this out in six phases.CLAUDE.mdat both docs, so agents don't trustnpm run test:perfnumbers.Docs only; no code changes.
Test plan
tests/perf.mjsto confirm the flag-parsing bugs and the monotonic RSS readings that the review describes.npm testpasses (pre-commit hook).🤖 Generated with Claude Code