Add Docker track-race integration tests (`it/tracks`) with Compose utilities and CI by fressi-elastic · Pull Request #2096 · elastic/rally

fressi-elastic · 2026-04-09T03:30:35Z

Summary

This introduces a matrix of test cases that run rally race command agains ES distributions 8.x and 9.x with all the tracks from the public repository. For each track/ES version combination it runs an integration test that it executes rally and ES in an isolated environment orchestrated on top of docker compose. It introduces two CI jobs, one for each major ES version. When used locally by default it runs agains both ES versions in parallel for each track.

Goal

Provide a very fast gating test suiteto spot integration issues between rally, pinned ES versions and rally tracks repository. The purpose is to leverage engineer from unreliable and constly manual testing when proposing potentially disrupting changes that elsewhere would require long and expensive end to end testing which would involve using esbench and other potentially failing components. This introduces a gating pipeline intended to avoid errors in rally and rally tracks repository to be merged undetected and pollute production environment.

Time execution optimizations

Because rally race command could take very long time it uses the following methods:

it doesn't pretend to be an en-to-end test suits, it instead uses docker compose for isolating integration components. This allows to use the same VM to run the whole test suit (in parallel eventually).
it allows executing test cases in parallel using pytest-xdist plugin. In case local resources allow, it could run and arbitrary number of instances of the test runner (2 by default);
it timeouts rally race condition after a timeout period calculated by dividing a configurable amount of time by the number of scheduled test cases to be executed, to ensure the execution will not take more than the given total timeout. In case a race exceed this timeout without errors, it will consider the test case as succeeded. This is intentional because it expects integration failures should be handled by error handling of both client/server side.
it run races in testing mode when available
by default it skips a selected number of tracks know to be having problems to handle known issues in the tracks repository master branch

Introduce esrally.utils.compose for Docker Compose and Rally-in-container workflows, including ComposeConfig, run_compose wrappers, and list_tracks that parses tabulate-formatted rally list tracks stdout into a list of row dicts. Fix Config.from_config to pass copy_from so wrapped configs inherit options. Extend types.Section and types.Key for the compose config section so mypy accepts the new settings. Add table-driven tests in tests/utils/compose_test.py using esrally.utils.cases and dataclass case specs. Made-with: Cursor

Wire compose.rally_race with an optional challenge argument forwarded to rally race. Add it/tracks_test.py to exercise Rally in Docker against compose-managed Elasticsearch across inlined TrackCase definitions. Extend compose unit tests accordingly. Made-with: Cursor

Ship a default compose.yaml next to the compose helpers and resolve its build context to the Rally project root so docker/Dockerfiles/dev/Dockerfile COPY paths work. Default compose.dir is the parent of the esrally package; run_compose injects RALLY_COMPOSE_DIR for Compose interpolation and accepts an optional env overlay. RALLY_COMPOSE_FILE overrides the default compose file path. Tests cover the new defaults and env wiring. Made-with: Cursor

- Ignore only repository-root tracks/ so it/tracks/ is not excluded - Rename it/tracks_test.py to it/tracks/race_test.py - Add README for Docker-based race integration tests and how to run them Made-with: Cursor

Add TrackCase.skip_reason_by_es_version and pytest.skip in test_race_with_track for fourteen failing (track, ES_VERSION) pairs documented in TRACK_RACE_EXECUTION_FINDINGS.md (pip deps, Netty setting, track load, etc.). Document the findings file and skip field in it/tracks/README.md. Made-with: Cursor

Add it/tracks conftest with CLI/env for ES versions, total timeout budget, track-name fnmatch deselection (pytest_deselected), and no-skip for skip_reason_by_es_version. Parametrize elasticsearch indirectly; stash race_timeout_s as total_min*60/max(1,N) where N excludes version-skip rows unless IT_TRACKS_NO_SKIP/--it-tracks-no-skip. run_service(remove=True) always remove_service(--stop --force) in finally so subprocess timeouts do not orphan compose run containers; extend tests. Document in it/tracks/README.md and Makefile; add helpers unit tests. Made-with: Cursor

docker compose rm/stop exclude one-off run containers (Compose oneOffExclude), so the previous finally-block rm --stop never removed rally-run-* orphans after subprocess timeouts. After compose run, run compose kill (includes one-offs), compose ps -a -q, then docker rm -f on listed IDs. Document in it/tracks/README and remove_service docstring; update compose tests. Made-with: Cursor

Run track races in parallel with pytest-xdist: one worker per configured Elasticsearch version, loadgroup scheduling, COMPOSE_PROJECT_NAME per worker, and race timeout scaled by worker count. Controller builds the Rally image once; workers tear down compose on session end. Add compose.teardown_project and related tests. Shorten elasticsearch parametrization ids from es_version_<ver> to es_<ver> for clearer nodeids. Made-with: Cursor

Read PYTEST_XDIST_WORKER_COUNT in it/tracks helpers so per-version xdist grouping is omitted when the pool is larger than the ES version list; worker subprocesses clear config numprocesses, so the env var is required for that logic to apply during collection. Teach run_compose to pass --project-name when COMPOSE_PROJECT_NAME is set, unless compose_options already sets -p/--project-name; extend compose tests and note behavior in it/tracks README. Build the Rally service from it/tracks/Dockerfile with RALLY_TRACKS_REF in compose.yaml. Made-with: Cursor

Add a matrix job on ubuntu-22.04 that runs make it_tracks with ARGS=-n 4 and IT_TRACKS_ES_VERSIONS per matrix cell, mirroring rally-tracks-compat setup (disk space, timeouts, Slack on failure). Cross-reference the ES version list in it/tracks/helpers.py. Also clarify it/tracks README (prerequisites, make it vs it_tracks, session teardown) and tidy race_test module docstrings. Made-with: Cursor

- Reformat tests/utils/compose_test.py so pre-commit black passes on CI. - Use cgr.dev/chainguard/wolfi-base for it/tracks Docker build so anonymous GitHub Actions runners can pull the base (docker.elastic.co wolfi requires auth). Made-with: Cursor

… artifact - Bind-mount ES and Rally logs to checkout logs/ with per-test paths from nodeid (:: split into path segments for Compose-safe bind sources). - Set COMPOSE_PROJECT_NAME from a Docker-safe slug per pytest nodeid; drop xdist worker-only project name. - rally_race: capture merged stdout+stderr and log on nonzero exit; pytest.ini INFO logging with explicit format for failure reports. - compose.yaml: ES_LOG_STYLE=file, IT_TRACKS_HOST_LOG_DIR default for unset env. - CI: zip logs/ and upload-artifact on it-tracks-race failure; gitignore /logs/. - Tests and README updated; pylint: top-level rally_root imports. Made-with: Cursor

When the nodeid file segment is basename-only (confcutdir), resolve IT_TRACKS_HOST_LOG_DIR under logs/it/tracks/<module>/ instead of logs/<module>/. Document in README; add unit test. Made-with: Cursor

TestLiteralArgs walks Python files under the repo; it/tracks host logs can include directories named like race_test.py under logs/, which matched the glob and broke read_text(). Exclude paths under checkout logs/. Made-with: Cursor

- conftest: tryfirst pytest_runtest_teardown writes merged docker compose logs to logs/.../containers.log before elasticsearch fixture tears down es01. - compose: add write_project_logs; run_compose skips default stderr=PIPE when capture_output=True (subprocess.run compatibility). - rally_race: force remove=False so compose run rally omits --rm; one-off is removed via teardown_project after logs, so containers.log includes rally. - Makefile: document clean-others removing logs/. - tests: compose coverage for logs capture, capture_output, remove=False path. Made-with: Cursor

- Replace snapshot compose logs with spawn_compose_logs_follow; autouse fixture tails logs into containers.log while elasticsearch is up. - run_service accepts name for docker compose run --name; run_rally defaults to rally; strip duplicate --name in compose_options. - rally_race forwards **kwargs to run_rally (e.g. name= for overrides). - Extend compose tests; call rally_race with explicit kwargs for mypy. - Document short container name and cleanup in README; clarify race_test docstring. Made-with: Cursor

run_rally uses rally_ plus eight URL-safe base64 chars when name is omitted, avoiding container name clashes under parallel pytest-xdist. Tests patch the default for stable argv checks; add format assertion. README documents the default and name= override. Made-with: Cursor

Pre-create es01/ and rally/ under IT_TRACKS_HOST_LOG_DIR and chmod 777 so Elasticsearch (uid 1000) can write gc.log and logs when pytest runs as a different user or Docker would create missing mount paths as root on Linux. Made-with: Cursor

Reduces parallel Docker/Rally load per job by changing ARGS from -n 4 to -n 1. Made-with: Cursor

- Dockerfile: chown bundled rally-tracks to 1000:0; remove chmod -R a-w so the track preparator can write under /tracks in the IT image. - race_test: represent per-ES skips as ordered (prefix, reason) lists; resolve skips via helpers alongside pytest. - helpers: add it_tracks_skip_reason_for_entries and it_tracks_skip_reasons_enabled (env uses convert.to_bool with explicit on); race_item_counts_toward_timeout_budget shares the same prefix matching. - conftest: derive timeout N from the same skip-reason rules as the race test. - README: document skip map semantics and helpers. - tests: relocate to tests/it/tracks/helpers_test.py and extend cases. Made-with: Cursor

…races Set CFLAGS/CXXFLAGS so vendored trec_eval in pytrec_eval==0.5 compiles under GCC on Wolfi (gnu89 and relaxed incompatible-pointer diagnostics). Pre-install pytrec_eval and numpy into the Rally venv alongside esrally. Drop Docker IT skip reasons for search/mteb/dbpedia, msmarco-v2-vector, and msmarco-passage-ranking; run dbpedia in test mode like other default tracks. Run the sql track with default test_mode (True). Made-with: Cursor

… tags Document the completed make it_tracks run (three tracks, 8.19.14/9.3.3): 6 passed in ~21 minutes. Align default Elasticsearch version wording with helpers.DEFAULT_IT_TRACKS_ES_VERSIONS; label the 14-row matrix as historical; note IT Dockerfile in infrastructure section. Made-with: Cursor

Copilot

Pull request overview

Adds a new Docker-based integration test suite under it/tracks that runs esrally race against an Elasticsearch node from a bundled Compose stack, plus shared Compose utilities and CI wiring to execute these races across ES versions.

Changes:

Introduces it/tracks integration tests (pytest options, xdist grouping, per-test Compose project isolation, host-mounted logs, Docker image + Compose stack).
Adds esrally.utils.compose helpers for Compose execution, one-off container cleanup, and Rally/Elasticsearch wrappers, with unit tests.
Updates developer tooling/CI (pytest-xdist dependency, Makefile targets, gitignore, GitHub Actions job + log artifact upload on failure).

Reviewed changes

Copilot reviewed 20 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Adds locked deps for `pytest-xdist` (and `execnet`).
`pyproject.toml`	Adds `pytest-xdist` to `develop` extras.
`Makefile`	Adds `it_tracks` / `it_tracks_image` / `clean-it_tracks` targets; excludes `it/tracks` from `make it` by default.
`.gitignore`	Ignores repo-root `logs/` for host-mounted it/tracks logs; scopes `/tracks/` ignore to repo root.
`.github/workflows/ci.yml`	Adds `it-tracks-race` job matrix to run track races per ES version and upload zipped logs on failure.
`it/tracks/__init__.py`	Declares `it.tracks` as a package for helpers/tests.
`it/tracks/pytest.ini`	Configures isolated pytest settings for `it/tracks` (confcutdir behavior + xdist defaults + log capture).
`it/tracks/helpers.py`	Implements pure helpers for CLI/env parsing, xdist worker/group decisions, log path mapping, and timeout math.
`it/tracks/conftest.py`	Registers it/tracks-specific pytest options, parametrization, deselection, timeout stashing, and compose log following.
`it/tracks/race_test.py`	Adds parametrized Docker-based `rally race` integration test across tracks and ES versions with skip logic + timeouts.
`it/tracks/compose.yaml`	Defines Compose stack for ES + Rally services with bind-mounted logs and shared Rally image tag.
`it/tracks/Dockerfile`	Builds a dedicated Rally driver image for it/tracks (Wolfi base, installs deps, clones rally-tracks).
`it/tracks/README.md`	Documents how to run the suite, options/env vars, xdist behavior, timeouts, and cleanup strategy.
`it/tracks/TRACK_RACE_EXECUTION_FINDINGS.md`	Captures observed track/version failures and operational notes to inform skip rules and debugging.
`esrally/utils/compose.py`	Adds Compose orchestration utilities (run/build/up/down, log follow, rally wrappers, one-off cleanup).
`esrally/utils/resources/compose.yaml`	Existing Compose resource referenced by the new Compose utilities/config behavior.
`esrally/types.py`	Adds `compose.*` keys/modules to the allowed config/type lists.
`esrally/config.py`	Adjusts `Config.from_config()` to copy values via `copy_from` when wrapping a `types.Config`.
`tests/utils/compose_test.py`	Adds comprehensive unit tests for `esrally.utils.compose` helpers and behaviors.
`tests/types_test.py`	Excludes generated `logs/` tree from type/literal-arg scanning to avoid false positives.
`tests/it/tracks/helpers_test.py`	Adds unit tests for `it.tracks.helpers`.
`tests/it_tracks_helpers_test.py`	Adds (duplicate) unit tests for `it.tracks.helpers` at repo root test path.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 18 out of 20 changed files in this pull request and generated 9 comments.

Compose stack code is only used by track race integration tests; colocate it with those tests and drop the esrally.utils unit tests. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 3 comments.

Compose helpers are IT-only: use fixed paths and COMPOSE_COMMAND instead of rally.ini, and remove ComposeConfig and global init_config. Delete it/tracks/README.md; conftest.py documents pytest options. Co-authored-by: Cursor <cursoragent@cursor.com>

Restore tests/ to match origin/master so the PR focuses on it/tracks integration tests and compose helpers. Co-authored-by: Cursor <cursoragent@cursor.com>

Drop list_tracks and parse_tabulate_simple_table with no callers, and remove unused challenge support from rally_race and TrackCase. Co-authored-by: Cursor <cursoragent@cursor.com>

Mount es_data at /usr/share/elasticsearch/data so Elasticsearch uses the named volume and teardown can remove cluster data reliably. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep rally-tracks at master for integration signal; document build-arg override for reproducible and future cross-repo gating CI jobs. Co-authored-by: Cursor <cursoragent@cursor.com>

Prefix the docker compose down recipe with `-` so clean-others and clean-docs still run when Docker is missing or the daemon is down. Co-authored-by: Cursor <cursoragent@cursor.com>

fressi-elastic · 2026-05-21T07:39:11Z

Task 4 (make it pytest paths): Verified make -n it already passed it/ and --ignore=it/tracks as separate pytest argv entries via recursive $(MAKE) test ARGS=….

make it ARGS='-k foo' replaced the default paths entirely ($(or $(ARGS), …)), so extra args did not append.

Applied Option A: it now invokes pytest directly (same log setup as test, mirror it_tracks style):

```makefile
it: venv
mkdir -p ...
uv run -- pytest ... it/ --ignore=it/tracks $(ARGS)
```

Dry-run: make -n it ARGS='-k foo' → … it/ --ignore=it/tracks -k foo.

Recursive $(MAKE) test with ARGS="$(or …)" already split it/ and --ignore=it/tracks, but overriding ARGS replaced those defaults. Mirror it_tracks: call pytest with fixed paths and append $(ARGS). Co-authored-by: Cursor <cursoragent@cursor.com>

fressi-elastic · 2026-05-21T07:45:40Z

Task 4 fix pushed in 0a916bc — make it invokes pytest directly with it/ --ignore=it/tracks and appends $(ARGS). Makefile L210 review thread marked resolved.

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 4 comments.

+    """True if ``compose_options`` already passes ``-p`` / ``--project-name`` to the CLI."""
+    if not compose_options:
+        return False
+    return any(opt in ("-p", "--project-name") for opt in compose_options)
+
+


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Update defaults to 8.19.15 and 9.3.4, extend the it-tracks-race CI matrix with 9.4.1, and align the compose image fallback tag. Co-authored-by: Cursor <cursoragent@cursor.com>

Running containers were not stopped before docker compose rm, which could leave stale es01 state and cause flaky integration tests. Co-authored-by: Cursor <cursoragent@cursor.com>

fressi-elastic added 10 commits April 6, 2026 01:45

it: organize track race tests under it/tracks

3e4a7dd

- Ignore only repository-root tracks/ so it/tracks/ is not excluded - Rename it/tracks_test.py to it/tracks/race_test.py - Add README for Docker-based race integration tests and how to run them Made-with: Cursor

fressi-elastic changed the title ~~It/tracks~~ CI: run it/tracks Docker races per Elasticsearch version (4 workers) Apr 9, 2026

fressi-elastic mentioned this pull request Apr 9, 2026

Create integration test for rally and many tracks. #2052

Closed

fressi-elastic changed the title ~~CI: run it/tracks Docker races per Elasticsearch version (4 workers)~~ Add Docker track-race integration tests (it/tracks) with Compose utilities and CI Apr 9, 2026

fressi-elastic added 11 commits April 9, 2026 10:35

it/tracks: put host logs under logs/it/tracks/ for short pytest nodeids

580454f

When the nodeid file segment is basename-only (confcutdir), resolve IT_TRACKS_HOST_LOG_DIR under logs/it/tracks/<module>/ instead of logs/<module>/. Document in README; add unit test. Made-with: Cursor

ci: run it/tracks tests with a single pytest-xdist worker

3b2b042

Reduces parallel Docker/Rally load per job by changing ARGS from -n 4 to -n 1. Made-with: Cursor

fressi-elastic requested a review from Copilot April 10, 2026 06:49

Copilot started reviewing on behalf of fressi-elastic April 10, 2026 06:50 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

Comment thread it/tracks/helpers.py Outdated

Comment thread it/tracks/race_test.py

Comment thread tests/it_tracks_helpers_test.py Outdated

fressi-elastic added 2 commits April 10, 2026 10:07

sql track don't support test mode

5466629

Merge branch 'master' of github.com:elastic/rally into it/tracks

472f06a

fressi-elastic added 2 commits May 20, 2026 10:50

Update expected failures documents.

7754b76

Merge branch 'master' of github.com:elastic/rally into it/tracks

97c630f

fressi-elastic marked this pull request as ready for review May 20, 2026 08:53

fressi-elastic requested a review from a team as a code owner May 20, 2026 08:53

Reduce the size of the PR.

1b9912e

fressi-elastic requested a review from Copilot May 20, 2026 09:12

Copilot started reviewing on behalf of fressi-elastic May 20, 2026 09:13 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

fressi-elastic requested a review from Copilot May 20, 2026 12:20

Copilot started reviewing on behalf of fressi-elastic May 20, 2026 12:21 View session

Move Docker compose helpers into it/tracks.

1c0ac1b

Compose stack code is only used by track race integration tests; colocate it with those tests and drop the esrally.utils unit tests. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI reviewed May 20, 2026

View reviewed changes

Comment thread it/tracks/compose.yaml Outdated

Comment thread Makefile

Comment thread it/tracks/Dockerfile

fressi-elastic and others added 8 commits May 20, 2026 14:43

Drop tests/ changes from the it/tracks branch.

ed17f52

Restore tests/ to match origin/master so the PR focuses on it/tracks integration tests and compose helpers. Co-authored-by: Cursor <cursoragent@cursor.com>

Remove unused it/tracks compose and challenge code.

9bb3dae

Drop list_tracks and parse_tabulate_simple_table with no callers, and remove unused challenge support from rally_race and TrackCase. Co-authored-by: Cursor <cursoragent@cursor.com>

Restore unit tests from upstream master.

41c4826

restore esrally to upstream master

6b25dcc

Fix it/tracks ES data volume mount path.

af2c924

Mount es_data at /usr/share/elasticsearch/data so Elasticsearch uses the named volume and teardown can remove cluster data reliably. Co-authored-by: Cursor <cursoragent@cursor.com>

Document RALLY_TRACKS_REF default master and override for it/tracks.

f07b2a8

Keep rally-tracks at master for integration signal; document build-arg override for reproducible and future cross-repo gating CI jobs. Co-authored-by: Cursor <cursoragent@cursor.com>

Make clean-it_tracks best-effort so make clean does not require Docker.

f3eee8a

Prefix the docker compose down recipe with `-` so clean-others and clean-docs still run when Docker is missing or the daemon is down. Co-authored-by: Cursor <cursoragent@cursor.com>

fressi-elastic requested a review from Copilot May 21, 2026 07:53

Copilot started reviewing on behalf of fressi-elastic May 21, 2026 07:53 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

fressi-elastic and others added 3 commits May 21, 2026 10:16

Potential fix for pull request finding

f176d29

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Bump it/tracks Elasticsearch image versions.

5be7628

Update defaults to 8.19.15 and 9.3.4, extend the it-tracks-race CI matrix with 9.4.1, and align the compose image fallback tag. Co-authored-by: Cursor <cursoragent@cursor.com>

Stop es01 before compose rm in race_test ES fixture teardown.

f362a4f

Running containers were not stopped before docker compose rm, which could leave stale es01 state and cause flaky integration tests. Co-authored-by: Cursor <cursoragent@cursor.com>

pquentin removed the request for review from a team May 28, 2026 13:49

Uh oh!

Conversation

fressi-elastic commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Goal

Time execution optimizations

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fressi-elastic commented May 21, 2026

Uh oh!

fressi-elastic commented May 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fressi-elastic commented Apr 9, 2026 •

edited

Loading