Benchmarks#60
Open
yisraelU wants to merge 7 commits into
Open
Conversation
- CommandBenchmark: end-to-end throughput/latency for set/get, batches, parallel pipelines, incr/decr, hset/hget, lpush/lpop, sadd/smembers - CodecBenchmark: encode/decode at short/medium/large sizes Usage: sbt "benchmarks/Jmh/run .*CommandBenchmark" Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runs JMH benchmarks on main after tests pass. Outputs formatted results to GitHub Step Summary and uploads JSON artifact for trend tracking. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new JMH-based benchmarking module and wires it into CI so the project can measure codec and command performance against a running Valkey instance.
Changes:
- Introduces
benchmarkssbt subproject usingsbt-jmh. - Adds JMH benchmark suites for command execution and codec encode/decode.
- Adds a GitHub Actions job (main-branch only) to run benchmarks and publish results.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| modules/benchmarks/src/main/scala/dev/profunktor/valkey4cats/benchmarks/CommandBenchmark.scala | Adds JMH benchmarks for common Valkey command patterns (set/get, batches, counters, hashes, lists, sets). |
| modules/benchmarks/src/main/scala/dev/profunktor/valkey4cats/benchmarks/CodecBenchmark.scala | Adds JMH benchmarks for UTF-8 codec encode/decode and round-trips over different payload sizes. |
| build.sbt | Adds a new benchmarks module configured with JmhPlugin and noPublish. |
| .github/workflows/ci.yml | Adds a benchmark job on main to run JMH, summarize results, and upload JSON output. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| run: | | ||
| echo "## Benchmark Results" >> $GITHUB_STEP_SUMMARY | ||
| echo '```' >> $GITHUB_STEP_SUMMARY | ||
| nix run .#sbt -- "benchmarks/Jmh/run -i 5 -wi 3 -f 0 -t 1 .*CodecBenchmark" 2>&1 | grep -E "^(Benchmark|Codec)" >> $GITHUB_STEP_SUMMARY || true |
Comment on lines
+42
to
+45
| @Benchmark | ||
| def setBatch10(): Unit = | ||
| (1 to 10).toList.traverse_(i => valkey.set(s"bench:batch:$i", "v").void) | ||
| .unsafeRunSync() |
Comment on lines
+47
to
+50
| @Benchmark | ||
| def getBatch10(): Unit = | ||
| (1 to 10).toList.traverse_(i => valkey.get(s"bench:batch:$i").void) | ||
| .unsafeRunSync() |
Comment on lines
+52
to
+55
| @Benchmark | ||
| def pipeline100(): Unit = | ||
| (1 to 100).toList.parTraverse_(i => valkey.set(s"bench:pipe:$i", "v").void) | ||
| .unsafeRunSync() |
| @Fork(1) | ||
| class CommandBenchmark: | ||
|
|
||
| given Log[IO] = Log.Stdout.instance[IO] |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Use Log.NoOp instead of Log.Stdout to avoid noisy output - Precompute index lists (indices10, indices100) to avoid allocation in hot path - Rename pipeline100 → parallelSet100 to reflect parTraverse semantics - Fix CI grep pattern to match fully-qualified JMH output lines Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment on lines
+71
to
+83
| python3 -c " | ||
| import json, sys | ||
| data = json.load(open('results.json')) | ||
| print(f\"{'Benchmark':<40} {'Mode':<6} {'Score':>10} {'Error':>10} {'Units'}\") | ||
| print('-' * 80) | ||
| for b in data: | ||
| name = b['benchmark'].split('.')[-1] | ||
| mode = b['mode'] | ||
| score = f\"{b['primaryMetric']['score']:.3f}\" | ||
| error = f\"± {b['primaryMetric']['scoreError']:.3f}\" if b['primaryMetric']['scoreError'] else '' | ||
| unit = b['primaryMetric']['scoreUnit'] | ||
| print(f\"{name:<40} {mode:<6} {score:>10} {error:>10} {unit}\") | ||
| " >> $GITHUB_STEP_SUMMARY |
| name = b['benchmark'].split('.')[-1] | ||
| mode = b['mode'] | ||
| score = f\"{b['primaryMetric']['score']:.3f}\" | ||
| error = f\"± {b['primaryMetric']['scoreError']:.3f}\" if b['primaryMetric']['scoreError'] else '' |
Comment on lines
+50
to
+53
| @Benchmark | ||
| def getBatch10(): Unit = | ||
| indices10.traverse_(i => valkey.get(s"bench:batch:$i").void) | ||
| .unsafeRunSync() |
| def setGet(): String = | ||
| (valkey.set("bench:key", "value") *> valkey.get("bench:key")) | ||
| .unsafeRunSync() | ||
| .fold(_ => "", _.getOrElse("")) |
Comment on lines
+77
to
+80
| @Benchmark | ||
| def saddSmembers(): Unit = | ||
| (valkey.sadd("bench:set", "a", "b", "c") *> | ||
| valkey.smembers("bench:set")).void |
- Fail benchmark on Valkey errors instead of swallowing them - Pre-populate keys in @setup so getBatch10 measures hits, not misses - Use per-invocation keys in saddSmembers so SADD is never a no-op - Fix CI python script: use heredoc to avoid indentation errors - Fix scoreError check: use `is not None` instead of truthiness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment on lines
+23
to
+26
| private var valkey: ValkeyCommands[IO, String, String] = uninitialized | ||
| private var cleanup: IO[Unit] = uninitialized | ||
| private var invocationCount: Long = 0L | ||
|
|
| uses: actions/checkout@v4 | ||
|
|
||
| - name: 🚀 Start Valkey | ||
| run: docker compose up -d |
Comment on lines
+46
to
+48
| needs: build | ||
| if: github.ref == 'refs/heads/main' | ||
| steps: |
- Use AtomicLong for invocationCount (thread-safe under -t > 1) - Start only SingleNode in benchmark CI job (saves resources) - Compile benchmarks on PRs to catch breakage early Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment on lines
+51
to
+54
| unwrap( | ||
| (valkey.set("bench:key", "value") *> valkey.get("bench:key")) | ||
| .unsafeRunSync() | ||
| ).getOrElse("") |
Comment on lines
+57
to
+60
| def setBatch10(): Unit = | ||
| indices10.traverse_(i => valkey.set(s"bench:batch:$i", "v").void) | ||
| .unsafeRunSync() | ||
|
|
|
|
||
| @Benchmark | ||
| def getBatch10(): Unit = | ||
| indices10.traverse_(i => valkey.get(s"bench:batch:$i").void) |
Comment on lines
+66
to
+70
| @Benchmark | ||
| def parallelSet100(): Unit = | ||
| indices100.parTraverse_(i => valkey.set(s"bench:par:$i", "v").void) | ||
| .unsafeRunSync() | ||
|
|
Comment on lines
+73
to
+74
| (valkey.incr("bench:counter") *> valkey.decr("bench:counter")).void | ||
| .unsafeRunSync() |
Comment on lines
+78
to
+79
| (valkey.hset("bench:hash", Map("field1" -> "val1")) *> | ||
| valkey.hget("bench:hash", "field1")).void |
Comment on lines
+84
to
+85
| (valkey.lpush("bench:list", "item") *> | ||
| valkey.lpop("bench:list")).void |
Comment on lines
+88
to
+94
| @Benchmark | ||
| def saddSmembers(): Unit = | ||
| val key = s"bench:set:${invocationCount.getAndIncrement()}" | ||
| (valkey.sadd(key, "a", "b", "c") *> | ||
| valkey.smembers(key) *> | ||
| valkey.del(key)).void | ||
| .unsafeRunSync() |
Comment on lines
+87
to
+90
| echo "### Codec Benchmarks" >> $GITHUB_STEP_SUMMARY | ||
| echo '```' >> $GITHUB_STEP_SUMMARY | ||
| nix run .#sbt -- "benchmarks/Jmh/run -i 5 -wi 3 -f 0 -t 1 .*CodecBenchmark" 2>&1 | grep -E "(Benchmark|CodecBenchmark)" >> $GITHUB_STEP_SUMMARY || true | ||
| echo '```' >> $GITHUB_STEP_SUMMARY |
Agent-Logs-Url: https://github.com/profunktor/valkey4cats/sessions/c1ee751e-03ea-4baa-9715-276ebd7c3f24 Co-authored-by: yisraelU <4328266+yisraelU@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.