Summary
The Unit tests - IceTray (v1.13.0 - 3.10) CI job currently fails for every run — new PRs and re-runs of old commits alike. Test collection crashes before any test executes:
SystemError: initialization of _MuonGun raised unreported exception
A module that was compiled using NumPy 1.x cannot be run in NumPy 2.2.6 ...
AttributeError: _ARRAY_API not found
ImportError: numpy.core._multiarray_umath failed to import
The last green run of this job was 2026-07-18. The breakage is time-dependent, not commit-dependent: re-running the (previously SSL-flaky) job on the old merge commit ddc54b10 now fails with the same error (run 28811940069), as does current PR CI (e.g. #879, run 30861490587).
Root cause
Install order in .github/actions/install/action.yml:
-
pip install .[...] resolves numpy 1.26.4, honoring graphnet's numpy>=1.22,<2.0 pin from setup.py. ✓
-
The final step pip install git+https://github.com/thoglu/jammy_flows.git (unpinned git HEAD) pulls mhealpy → healpy. The latest healpy 1.20.0 requires numpy ≥ 2. graphnet is not part of that second resolve, so pip only warns about the conflict and upgrades numpy:
Collecting healpy (from mhealpy>=0.3.2->jammy_flows==1.1.0)
Uninstalling numpy-1.26.4:
Successfully installed healpy-1.20.0 jammy_flows-1.1.0 mhealpy-0.3.7 numpy-2.2.6 torchdiffeq-0.2.5
-
The container image icecube/icetray:icetray-devel-v1.13.0-ubuntu22.04-2025-02-12 ships C extensions (_MuonGun, …) compiled against numpy 1.x, which cannot load under numpy 2.2.6.
A healpy release requiring numpy ≥ 2 landed between 2026-07-18 and 2026-08-03, which is why the job silently flipped from green to red with no change in this repository.
Proposed fix
Constrain the jammy_flows install so numpy stays below 2 within the same resolve, forcing pip to select a numpy-1-compatible healpy:
pip install --no-cache-dir "numpy>=1.22,<2.0" git+https://github.com/thoglu/jammy_flows.git
Longer term: pin healpy/jammy_flows to fixed versions instead of git HEAD, or move to an icetray image built against numpy 2.
Once the fix is on main, all open PRs go green automatically on their next CI run (PR workflows run against the merge with the base branch).
Summary
The
Unit tests - IceTray (v1.13.0 - 3.10)CI job currently fails for every run — new PRs and re-runs of old commits alike. Test collection crashes before any test executes:The last green run of this job was 2026-07-18. The breakage is time-dependent, not commit-dependent: re-running the (previously SSL-flaky) job on the old merge commit
ddc54b10now fails with the same error (run 28811940069), as does current PR CI (e.g. #879, run 30861490587).Root cause
Install order in
.github/actions/install/action.yml:pip install .[...]resolves numpy 1.26.4, honoring graphnet'snumpy>=1.22,<2.0pin fromsetup.py. ✓The final step
pip install git+https://github.com/thoglu/jammy_flows.git(unpinned git HEAD) pullsmhealpy→healpy. The latest healpy 1.20.0 requires numpy ≥ 2. graphnet is not part of that second resolve, so pip only warns about the conflict and upgrades numpy:The container image
icecube/icetray:icetray-devel-v1.13.0-ubuntu22.04-2025-02-12ships C extensions (_MuonGun, …) compiled against numpy 1.x, which cannot load under numpy 2.2.6.A healpy release requiring numpy ≥ 2 landed between 2026-07-18 and 2026-08-03, which is why the job silently flipped from green to red with no change in this repository.
Proposed fix
Constrain the jammy_flows install so numpy stays below 2 within the same resolve, forcing pip to select a numpy-1-compatible healpy:
pip install --no-cache-dir "numpy>=1.22,<2.0" git+https://github.com/thoglu/jammy_flows.gitLonger term: pin
healpy/jammy_flowsto fixed versions instead of git HEAD, or move to an icetray image built against numpy 2.Once the fix is on
main, all open PRs go green automatically on their next CI run (PR workflows run against the merge with the base branch).