[ML] Run QA and PyTorch suites concurrently in a single triggered build

[edsavage]

Summary

When both the QA (ml_cpp_pr) and PyTorch suites are requested on a PR, trigger a single downstream appex-qa build with a comma-separated QAF_TESTS_TO_RUN instead of two separate trigger steps.

Previously the two suites were triggered as independent downstream builds. Because the appex-qa-stateful-custom-ml-cpp-build-testing pipeline has skip_queued_branch_builds=true, two builds on the same branch collided and one suite was silently skipped. Emitting a single trigger with both markers lets the appex-qa generator fan them out into concurrent parallel jobs.

Changes

• .buildkite/pipeline.json.py: merge the run_qa_tests / run_pytorch_tests trigger blocks into one. QAF_TESTS_TO_RUN is built from the requested suites:
  • QA only → ml_cpp_pr
  • PyTorch only → pytorch_tests
  • both → ml_cpp_pr,pytorch_tests

The separate PyTorch allowlist-validation step is unaffected.

⚠️ Ordering dependency

This is Part B. It depends on Part A — the appex-qa generator change in elastic/qaf-tests#309 — which teaches the generator to split a comma-separated QAF_TESTS_TO_RUN into parallel jobs. qaf-tests#309 must merge first. Until then, requesting both labels would pass ml_cpp_pr,pytorch_tests to the old generator, which treats it as a single invalid pytest -m expression. Single-suite requests are unaffected.

Testing

• Local dry-runs of the generator confirm valid pipeline JSON and correct QAF_TESTS_TO_RUN for QA-only, PyTorch-only, and both-labels cases.
• End-to-end validation of concurrent jobs performed on a qaf-tests feature branch build.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[Copilot]

Pull request overview

This PR updates the ml-cpp PR Buildkite pipeline generator to avoid QA and PyTorch downstream build collisions by triggering one appex-qa downstream build when both suites are requested, using a comma-separated QAF_TESTS_TO_RUN so the downstream generator can fan out suites into parallel jobs.

Changes:

• Merge the QA and PyTorch downstream trigger logic into a single trigger step gated on run_qa_tests || run_pytorch_tests.
• Build QAF_TESTS_TO_RUN as ml_cpp_pr, pytorch_tests, or ml_cpp_pr,pytorch_tests depending on requested suites.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[edsavage]

addressed both of copilot's comments in 5f74468:

1. Comma-separated QAF_TESTS_TO_RUN override → duplicated suites (line 96): the env override is now split on commas into individual suites, and the final list is de-duplicated while preserving order (dict.fromkeys). So both-labels + an override of ml_cpp_pr,pytorch_tests now yields ml_cpp_pr,pytorch_tests rather than ...,pytorch_tests,pytorch_tests.

2. Misleading step label (line 97): renamed to "Upload QA/PyTorch tests runner pipeline" since the single trigger now covers QA and/or PyTorch. (The label is display-only — generate_step doesn't derive a key from it and nothing depends on it.)

Verified with local dry-runs: QA-only, PyTorch-only, both, both + a pytorch-inclusive override (de-dups), and a QA subset override — all produce the expected QAF_TESTS_TO_RUN.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.buildkite/pipeline.json.py:107

• For PyTorch-only runs (config.run_pytorch_tests true, config.run_qa_tests false) this change switches the runner upload to run_qa_tests.yml.sh. That script emits steps labeled/commit-statused as “QA Tests”, which is misleading and also changes the commit status context compared to the previous run_pytorch_tests.yml.sh behavior. Consider keeping run_pytorch_tests.yml.sh for the PyTorch-only case and using run_qa_tests.yml.sh only when QA is involved (QA-only or QA+PyTorch).

                # De-duplicate while preserving order so an override that already
                # includes pytorch_tests does not produce a duplicate suite.
                env["QAF_TESTS_TO_RUN"] = ",".join(dict.fromkeys(qa_suites))
                pipeline_steps.append(pipeline_steps.generate_step("Upload QA/PyTorch tests runner pipeline",
                                                                   ".buildkite/pipelines/run_qa_tests.yml.sh"))

[edsavage]

Addressed the low-confidence note from the latest Copilot review (re: PyTorch-only runs being labelled/commit-statused as "QA Tests").

Rather than partially reverting to run_pytorch_tests.yml.sh for the PyTorch-only case (which would reintroduce two code paths and drop the ES_BRANCH/STACK_VERSION forwarding that run_qa_tests.yml.sh provides), run_qa_tests.yml.sh now derives a descriptor from QAF_TESTS_TO_RUN and uses it for the step label, echo, and github_commit_status context:

• ml_cpp_pr -> Trigger Appex QA Tests
• pytorch_tests -> Trigger Appex PyTorch Tests
• ml_cpp_pr,pytorch_tests -> Trigger Appex QA + PyTorch Tests

Substring matching keeps it correct for marker expressions (e.g. ml_cpp_pr and not slow). Note these contexts are not required status checks on any branch, so the rename has no branch-protection impact. Verified via dry-run across all cases; emitted YAML parses.

[elastic-vault-github-plugin-prod]

💔 Some backports could not be created

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
✅	9.4
✅	9.5

Manual backport

To create the backport manually run:

backport --pr 3131

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

[elastic-vault-github-plugin-prod]

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 3131

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details