Request for feature: Cross-Versions benchmarking

[diesalbla]

For most Scala projects, when I submit a Pull Request and I have to evaluate the performance impact of a Pull Request, I often have to manually do the following process:

• Checkout each distinct version, with and without the changes.
• Run the sbt:jmh command, a same command, in each of them.
• Retrieve and store the results from each run.
• Arrange the results of both runs onto a spreadsheet, which I then use to show the relative error of each measure, and the observed improvement.

It may be desirable if sbt:jmh plugin could offer a command to carry ouy these operations directly, and were able to generate a single summary table, which included the score and error for each commit, and the relative difference between them.

Benchmark  Mode  Cnt Score(master)   Error  Score(branch) Error  Change  Units
Fili       thrpt 10         45000       42         42000     41    93.3 %  B/s

[guizmaii]

Why closing @diesalbla? Is it supported now? Did you find a solution?

[diesalbla]

@guizmaii This is an issue I opened a few years ago during the contributions with fs2, due to the small inconvenience of switching branches and keeping results. It is not something I am aiming at doing soon, and it did not seem right to burden sbt maintainers with it. Maybe all that all is needed is a shell script, to do all the switch branch, setup, compilation, run benchmarks and save results.

I have not found any solution to do this locally. I understand, though, that there are some CI workflows to run benchmarks on each commit or PR. That may alleviate the need for this.

Leaving open for now.