Skip to content

[CI] Fix the CI jobs that fail on the OSDC images - #4544

Merged
huydhn merged 1 commit into
mainfrom
osdc/fix-libs-env
Oct 10, 2026
Merged

huydhn merged 1 commit into
mainfrom
osdc/fix-libs-env

Conversation

@huydhn

@huydhn huydhn commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor
  • Install python3-dev in the procgen, safety-gymnasium, minari and sglang setups, because the OSDC image's Python has no headers and building torchrl fails on Python.h.

  • Stop the habitat setup from putting /usr/bin first in PATH for the rest of the script, because later pip3 calls then hit the system pip, which refuses to install on the Ubuntu 24.04 image. Also install libglib2.0-0, which OpenCV needs and that image lacks.

  • Install cmake in the vllm and sglang setups, because the pinned tensordict builds a C++ extension and the OSDC image has no cmake. vllm takes it from apt, because pip builds tensordict in an isolated environment where the cmake wheel's launcher cannot import its own package.

  • Move [Refactor] Bound GPU test parallelism and reap strays #4532's stray reaping into helpers/reap_strays.sh and run it on exit in the craftground and libero tests too, because their background Xvfb keeps the job's output open and the job hangs until it times out.

  • Treat the killed child as gone once it is a zombie in test_run_example_terminates_timeouts (folded in from [CI] Count a zombie as reaped in test_run_example_terminates_timeouts #4547). On OSDC the job container's PID 1 is tail -f /dev/null, which never reaps orphans, so os.kill(pid, 0) keeps succeeding on the zombie.

  • Map the container's hostname to loopback in the olddeps tests, as linux/scripts/run_all.sh already does, because the gloo and nccl transport tests in TestRayRB and test_inference_server.py fail with gaierror resolving it.

  • Install python3-venv in the vllm, sglang, openx and minari setups, the jobs that download from the HF Hub, because linux_job_v3's HF cache sync falls back to installing the AWS CLI in a venv and the Ubuntu 24.04 image can't create one, so ci-refresh-hf-cache runs never upload the models they download.

@pytorch-bot

pytorch-bot Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4544

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 456836b with merge base 48a06fc (image):

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 9, 2026
@github-actions github-actions Bot added the CI Has to do with CI setup (e.g. wheels & builds, tests...) label Oct 9, 2026
@huydhn huydhn added Environments/procgen Data/minari Environments/habitat Touches habitat code. Requires habitat run in PRs labels Oct 9, 2026
@huydhn
huydhn requested a review from atalman October 9, 2026 18:27
@huydhn
huydhn force-pushed the osdc/fix-libs-env branch from e0e7d6f to 4284cfd Compare October 9, 2026 18:30
@huydhn
huydhn changed the base branch from main to ci/pin-tensordict October 9, 2026 18:30
@huydhn
huydhn added this pull request to stack #4546 October 9, 2026 18:30
@huydhn huydhn added llm/ci Environments Adds or modifies an environment wrapper Data Data-related PR, will launch data-related jobs labels Oct 9, 2026
@huydhn
huydhn force-pushed the osdc/fix-libs-env branch 2 times, most recently from bc5dba6 to 36bb11e Compare October 9, 2026 18:37
@github-actions github-actions Bot added the Environments/craftground Triggers craftground (Minecraft) environment tests only label Oct 9, 2026
@huydhn
huydhn force-pushed the osdc/fix-libs-env branch from 36bb11e to 1cabb80 Compare October 9, 2026 18:45
@huydhn
huydhn marked this pull request as ready for review October 9, 2026 18:50
@huydhn
huydhn force-pushed the osdc/fix-libs-env branch 3 times, most recently from b82265c to 1bbc980 Compare October 9, 2026 23:19
@huydhn
huydhn force-pushed the osdc/fix-libs-env branch from 1bbc980 to ef9572c Compare October 10, 2026 00:01
Base automatically changed from ci/pin-tensordict to main October 10, 2026 03:37
@huydhn
huydhn force-pushed the osdc/fix-libs-env branch from ef9572c to f24617c Compare October 10, 2026 03:37
@huydhn huydhn added the ci/olddeps Run the tests-olddeps suite (oldest supported torch) on this PR label Oct 10, 2026
@huydhn
huydhn force-pushed the osdc/fix-libs-env branch from f24617c to ad40ed5 Compare October 10, 2026 04:27
@huydhn huydhn changed the title [CI] Fix the libs and habitat setup scripts on the OSDC images [CI] Fix the CI jobs that fail on the OSDC images Oct 10, 2026
@huydhn
huydhn force-pushed the osdc/fix-libs-env branch from ad40ed5 to 9d2e955 Compare October 10, 2026 07:01
@huydhn

huydhn commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

@pytorchbot drci

@huydhn
huydhn force-pushed the osdc/fix-libs-env branch from 9d2e955 to 456836b Compare October 10, 2026 08:22
@huydhn

huydhn commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

@pytorchbot drci

1 similar comment
@huydhn

huydhn commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

@pytorchbot drci

@huydhn
huydhn merged commit b295f5b into main Oct 10, 2026
178 of 184 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/olddeps Run the tests-olddeps suite (oldest supported torch) on this PR CI Has to do with CI setup (e.g. wheels & builds, tests...) ci-refresh-hf-cache CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Data/minari Data Data-related PR, will launch data-related jobs Environments/craftground Triggers craftground (Minecraft) environment tests only Environments/habitat Touches habitat code. Requires habitat run in PRs Environments/procgen Environments Adds or modifies an environment wrapper llm/ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants