Using an accurate benchmark tool

[waghanza]

We are currently using wrk. This tool is great, no doubt about that. However, the purpose of such a tool is to check how a framework could perform at its best (grab the maximum performance it could give).

This project aims to reflect how frameworks could perform in real world scenarios (for example, using constant throughput). For this reason, I'm considering switching this tool.

Candidates are :

• https://github.com/tsenart/vegeta
• https://github.com/tag1consulting/goose
• https://github.com/codesenberg/bombardier
• https://github.com/sharkdp/hyperfine

Goose seems accurate for me. Any suggestions @dominikzogg @jeremyevans @panesofglass @jodosha @timriley @ahopkins @appleboy @yusukebe @p8 @mcollina @fundon @ohler55 @sy-records @enghitalo @lukaszlenart ?

PS : The idea is to change this tool on Q2 2025

PSS : Sorry to ping some individuals, I do not know how to ping the whole team at once 😇

[waghanza]

Sorry : not hyperfine (more to benchmark cli tools)

[fundon]

https://github.com/hatoo/oha
It is also a good testing tool.

[waghanza]

Seems more like a PoC for me hatoo/oha#617

But to be honest, it's promising

[appleboy]

Yes, tsenart/vegeta is a versatile HTTP load testing tool that can handle real-world scenarios, including constant throughput, which is what you're looking for. Vegeta is designed to be both a command-line utility and a library, making it flexible for various use cases.

Key Features of Vegeta

1. Constant Throughput: Vegeta can maintain a constant request rate, which is useful for simulating real-world traffic patterns.
2. Ease of Use: It has a simple command-line interface and can be easily integrated into scripts and CI/CD pipelines.
3. Detailed Reporting: Vegeta provides detailed metrics and can generate reports in various formats, including JSON and HTML.
4. Flexibility: It supports multiple attack modes, including rate-limited and burst modes, and can handle complex scenarios with custom headers, bodies, and methods.

[waghanza]

Also, how can vegeta handle constant throughput ? I mean, how can we make an equal sign between constant throughput and a real world scenario, there is no control about throughout

[waghanza]

Vegeta use 50rps per default, how can we define the rate to use facing the scenario we use ?

[waghanza]

I have tested vegeta (without rate limit), just to compare to wrk (since it has not rate limit) , and I can not explain result I've found in tsenart/vegeta#720

[waghanza]

How can I use constant throughput in vegeta ?

How can I use rate param in vegeta ? Nad this it really accurate in benchmark context @appleboy

cc @akyoto

[akyoto]

How can I use constant throughput [...] Nad this it really accurate in benchmark context

Constant throughput is useful in private benchmarks at a company with very clear constraints.
Constant throughput should be avoided for public benchmarks.

[mcollina]

I think wrk is ok. wrk2 is better ;).

[waghanza]

I do not understand how using constant throughput could be helpful.

I mean, with constant throughput approach, we must know the rate we should use. With wrk2, the rate used (by default) is 1000, but how can we know that 1000 is accurate for our scenario ? Or define any rate for it ?

[waghanza]

wrk2 is not yet compilable for arm

[yusukebe]

I prefer to bombardier, though I don't have any concrete reason. My favorite benchmark by @SaltyAom is using it:

https://github.com/SaltyAom/bun-http-framework-benchmark

[waghanza]

I there a way to enable any rate limit with it @yusukebe @SaltyAom ?

This feature is important, imho, to make more realistic results

[alanmsant2]

I use rewrk (Rust) link

[waghanza]

Any rationales behind ?

[alanmsant2]

In my perception, I found the results more accurate with Rewrk

[panesofglass]

I've only used bombardier, and I thought it did a good job. I'll have to try the others, but I cannot offer any good comparisons.

[cyrusmsk]

In "Clients" folder some comparison of generator tools as well
https://github.com/dpbench/dpbench

[MDA2AV]

higher throughput than any other load generator and precise latency values

https://github.com/MDA2AV/gcannon

[Kaliumhexacyanoferrat]

Results in the leaderboard look very suspicious

Results seem to have sorted - they refactored the tests a few times but now they seem quite balanced. And the actual run on real potent hardware, similar to TFB.

[waghanza]

ah I see ... and rage 364K
very different that what we have https://web-frameworks-benchmark.netlify.app/compare?f=actix,rails,flask,rage

It's running on very different hardware, also different test types, I wouldn't expect results to be any similar

What is more important for me is the ratio (rails is by X less performant than actix ...) not absolute values @MDA2AV

[MDA2AV]

Yes, but the tests would have to be similar, for example if you compare the CRUD test with a simple hello world test, the CRUD test will exercise the frameworks' threadpool, database driver or async/await performance, while the simple hello world test is gated by the framework's I/O and parser, even inside parser some frameworks are faster with chunked encoding requests and different parts of the parsing logic. Even if the tests are similar, by testing on a machine with 128 cores the scaling might not be the same as on a machine with 16, I've seen many frameworks that can't scale beyond 12 cores

[p8]

If you look at the TFB results for Actix and Rails...

On the plaintext test Actix is about 50 times faster:

https://www.techempower.com/benchmarks/#section=data-r22&f=zhb20v-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-yyku7z-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-1ekf&test=plaintext

On the JSON test Actix is about 13 times faster:

https://www.techempower.com/benchmarks/#section=data-r22&f=zhb20v-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-yyku7z-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-1ekf&test=json

On the Multi query Actix is 4 times faster:

https://www.techempower.com/benchmarks/#section=data-r22&f=zhb20v-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-yyku7z-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-zik0zj-1ekf&test=query

[waghanza]

ℹ️

• TFB is archived (at least the repo)
• Rust is 158 -> https://github.com/TechEmpower/FrameworkBenchmarks/blob/523534bb61450e3522d775a749ad060753e26e3a/frameworks/Rust/actix/actix-http.dockerfile#L1
• Ruby is 3.4 -> https://github.com/TechEmpower/FrameworkBenchmarks/blob/523534bb61450e3522d775a749ad060753e26e3a/frameworks/Ruby/rails/rails.dockerfile#L1
• Rails is 8.0 -> https://github.com/TechEmpower/FrameworkBenchmarks/blob/523534bb61450e3522d775a749ad060753e26e3a/frameworks/Ruby/rails/Gemfile#L3

[waghanza]

Maybe k6 is more accurate for what we do

[floatdrop]

Tried it in cloud, but k6 is a resource hog and needs a lot of cpu.

[waghanza]

Actually k6 eats some CPU, so benchmark figures will not be relevant. However :

• benchmark figure is not a good indicator -> relative figure (percents) will be more accurate for us
• in a near future, client will be separated from servers, so CPU eating in client side will not be a problem anymore

[floatdrop]

in a near future, client will be separated from servers, so CPU eating in client side will not be a problem anymore

Well, it will affect the cost of benchmark run (I was forced to spin up multiple k6 instances to achieve desired throughput) and it was biting my wallet.

[floatdrop]

I have published a zrk – wrk2 port to Zig with machine readable reports and TUI. You can check it out.

[floatdrop]

@waghanza better late, than never: little follow up – #9668