Repository navigation
Add goals_perftest and write down what loads the database - #25
Merged
Merged
Conversation
Builds a graph of goals - independent ones, a chain, or the butterfly network from the perftest branch - commits it, starts busy or threaded workers and reports: - time, goals/s and ms per dependency level, and with --handler-time (handlers sleeping like waiting for a remote API) the share of the throughput of workers always busy, - worker CPU and what worker connections wait for, sampled from pg_stat_activity, - database cost per goal in units that depend little on hardware: statements, transactions, WAL, buffer pages and rows, plus CPU time of server processes when the server runs locally. Per-statement numbers come from pg_stat_statements, if installed. The idle scenario reports the same per second of an idle worker. It lives in example_app, so it isn't shipped with the package. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gr3xEoHp8cwWtEUknjcjsM
database-load.md has the cost per goal and per idle worker, measured with goals_perftest, how to estimate it for a given database, and findings. The most promising one, picking getting expensive with dead index entries at the head of the queue, goes to TODO to investigate, next to small fixes the measurements found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gr3xEoHp8cwWtEUknjcjsM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The database is the hardest part to scale, so this measures how much work goals and workers give it.
No library code changes.
./manage.py goals_perftest(inexample_app, not shipped) builds a graph of goals, commits it, starts busy or threaded workers and reports throughput, worker CPU, what worker connections wait for, and database cost per goal: statements, transactions, WAL, buffer pages, rows, and server CPU when the server is local.Scenarios: independent goals, a chain, the butterfly network from Butterfly network #14 (closed in favour of this), and idle workers.
database-load.mdhas the numbers, how to estimate the load for a given database, and findings.TODO gets the most promising one to investigate and small fixes the measurements found.
Some numbers (PostgreSQL 16 on the same 4-core machine, no-op handlers):
An idle busy worker runs 7 transactions per second.
Test plan
poetry run pytest(95 passed),poetry run mypy .,poetry run flake8pg_stat_statements🤖 Generated with Claude Code
https://claude.ai/code/session_01Gr3xEoHp8cwWtEUknjcjsM
Generated by Claude Code