Switch to using strict channel priority during RAPIDS builds

[vyasr]

RAPIDS conda packages currently do not install successfully when using strict channel priority. This has caused some difficulty for users in the past. strict channel priority also in general leads to faster solves. The reason that RAPIDS requires flexible channel priority is that there are some packages that have historically been published to both the rapidsai[-nightly] and conda-forge channels. Typically this occurred because RAPIDS needed specific versions/builds of packages that were either not yet available on conda-forge. However, in recent years we have moved to a much stronger reliance on building and maintaining conda-forge packages as needed, so most of the packages that we've done this for in the past (ucx, nccl) are now made regularly available on conda-forge and no longer updated on the rapidsai[-nightly] channel.

We should clean out the old packages in the rapidsai[-nightly] channel that prevent strict solving from working. Rather than removing them altogether, we can move them under a new label so that old versions could still be installed with that label installed (although in general installing old versions will be quite challenging without a fully specified environment lock file anyway due to how conda-forge's global pinnings move and other packages on there are released).

Strict build checklist

Strict priority should be enabled at build time for the following repositories.

2025-03-18: Currently blocked by old versions of various dependencies in rapidsai-nightly (that are now available in conda-forge)
Removal requested in rapidsai/ops#3877 Resolved

2025-03-24: rapidsai/gha-tools#153 to remove rapidsai channel from non-release builds, as there are some leftover older packages in that channel that break strict channel dependency resolution.

"Base" repos (high priority, likely to work with little effort):

• rmm refactor(rattler): enable strict channel priority for builds rmm#1867
• dask-cuda chore(conda): enable strict channel priority for conda build dask-cuda#1458
• kvikio chore(conda): enable strict channel priority for conda build kvikio#674
• ucx-py refactor(rattler): enable strict channel priority for builds ucx-py#1127
• ucxx refactor(rattler): enable strict channel priority for builds ucxx#398
• cumlprims_mg https://github.com/rapidsai/cumlprims_mg/pull/240
• ptxcompiler
• pynvjitlink feat(conda): port conda recipe to rattler-build pynvjitlink#137
• rapids-build-backend
• rapids-cmake no conda anymore: refactor(conda): remove unused conda package and associated CI resources rapids-cmake#810

High priority, may require some effort:

• cudf (refactor(rattler): enable strict channel priority for builds NVIDIA/cudf#18358)
• raft refactor(rattler): enable strict channel priority for builds NVIDIA/raft#2632
• cuvs refactor(rattler): enable strict channel priority for builds NVIDIA/cuvs#823
• cuml refactor(rattler): enable strict channel priority for builds NVIDIA/cuml#6546
• cucim Port all conda recipes to rattler-build & use strict channel priority cucim#864
• cugraph refactor(rattler): enable strict channel priority for builds cugraph#5035

Low priority, may have blockers:

• cuxfilter refactor(rattler): enable strict channel priority for builds cuxfilter#679
• cuspatial
• nx-cugraph feat(rattler): port conda build recipe to rattler build nx-cugraph#121
• cugraph-gnn feat(rattler): port conda build recipe to rattler build cugraph-gnn#178
• dependency-file-generator
• integration
• jupyterlab-nvdashboard
• rapids-metadata
• rapidsmpf

Update 2025-03-21 (@gforsyth)

--

Certain builds (I think exclusively on cuda 11.8 on x86) require a version of libcufile that is only in the nvidia channel. Since many (but not all) versions of libcufile are also available on conda-forge, it is not possible to enable strict channel priority for these builds.

Representative error message:

ExplainedDependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {MatchSpec("libcufile-dev==1.4.0.31=0"), MatchSpec("libcufile==1.4.0.31=0")}
Encountered problems while solving:
  - package libcufile-1.4.0.31-0 is excluded by strict repo priority
  - package libcufile-dev-1.4.0.31-0 is excluded by strict repo priority

Builds affected:

• kvikio (conda-cpp-build / 11.8.0, 3.10, amd64, rockylinux8)

Followup items

Once strict builds are enabled (and the tooling is updated accordingly), we can also enforce installing with strict channel priority in CI. We'll want to enable strict channel priority (and the correct channel list) in C++ tests, Python tests, and docs builds.

• Update gha-tools to set correct channels and enable strict channel priority
• Update rapids-dependency solver to insert correct channels depending on build type.

[raydouglass]

This is mostly documenting some of my tests for installing and running older versions of RAPIDS.

We also need to test arm64 installs because RAPIDS supported arm64 before many conda-forge packages did and we released those packages in our rapidsai conda channel.

This is the test script I used to check for import errors. It is not comprehensive. https://gist.github.com/raydouglass/ff100a114c2a370b68131af55959afc0

Test machine:

• Driver 550.78
• System CTK 12.3
• x86_64
• Ubuntu 22.04.4
• 2x Quadro RTX 8000
• Tests were run bare-metal until otherwise stated

Here are the conda list for each environment below: https://gist.github.com/raydouglass/5948d6cab3d3c9f29cc02533bb2b4d25

23.02

Solves & tested: mamba create -n rapids-23.02 python=3.10 cudatoolkit=11.8 rapids=23.02

22.02

Solved with mamba create -n rapids-22.02 python=3.9 cudatoolkit=11.5 rapids=22.02.

Test errored with:

Traceback (most recent call last):
  File "/home/rdouglass/workspace/snippets/test_rapids.py", line 2, in <module>
    import cudf
  File "/home/rdouglass/mambaforge/envs/rapids-22.02/lib/python3.9/site-packages/cudf/__init__.py", line 5, in <module>
    validate_setup()
  File "/home/rdouglass/mambaforge/envs/rapids-22.02/lib/python3.9/site-packages/cudf/utils/gpu_utils.py", line 20, in validate_setup
    from rmm._cuda.gpu import (
  File "/home/rdouglass/mambaforge/envs/rapids-22.02/lib/python3.9/site-packages/rmm/__init__.py", line 16, in <module>
    from rmm import mr
  File "/home/rdouglass/mambaforge/envs/rapids-22.02/lib/python3.9/site-packages/rmm/mr.py", line 14, in <module>
    from rmm._lib.memory_resource import (
  File "/home/rdouglass/mambaforge/envs/rapids-22.02/lib/python3.9/site-packages/rmm/_lib/__init__.py", line 15, in <module>
    from .device_buffer import DeviceBuffer
  File "rmm/_lib/device_buffer.pyx", line 1, in init rmm._lib.device_buffer
TypeError: C function cuda.ccudart.cudaStreamSynchronize has wrong signature (expected __pyx_t_4cuda_7ccudart_cudaError_t (__pyx_t_4cuda_7ccudart_cudaStream_t), got cudaError_t (cudaStream_t))

I think this is a system CTK issue since running the script in the original unmodified rapidsai/rapidsai:22.02-cuda11.5-runtime-ubuntu20.04-py3.9 image works for cudf/cuml. I did not reinstall the rapids package in the container.

0.10

This is the first version with the rapids meta package.

Solves with mamba create -n rapids-0.10 python=3.6 cudatoolkit=9.2 rapids=0.10

I did not test this.

[vyasr]

Now that we have an idea of what works, the next step is to figure out what could break with strict channel priority and packages removed. The approach I would follow is to run the same installation commands as above, but adding the --strict-channel-priority flag. For a first pass, a dry run should be sufficient. For each version of RAPIDS tested, inspect the output list of packages and find which ones are being installed from the rapidsai channel. If any of them are packages that we plan to remove from the rapidsai channel, we should add those to the command with a channel specifier, e.g. for ucx mamba create ... rapids=${VERSION} conda-forge::ucx. This will force the conda solver to pull the ucx package from conda-forge instead of rapidsai(-nightly). The dry runs should be sufficient to put together this list and evaluate what will solve. Once a complete list is compiled going back to 23.02 (selected since that was the last working version tested above, but we could go back further) then we should actually create the environments (no dry run) and run the test script posted above to see if any results change. I expect that the dry runs should tell us most of what we need to know though since unless there are incompatible binaries of the same package with the same version on two different channels (hopefully unlikely) then if the solve succeeds we'll be getting package versions that should function according to the constraints in our package dependency spec.

[gforsyth]

• Testing package removals
• Results
  • With datashader=0.13.1a removed
  • With datashader=0.13.1a not removed
• Recommendations
  • Remove the packages listed above, including datashader (recommended)
  • Remove the packages listed above, don’t remove datashader and add all datashader versions >=0.14 to rapidsai
  • Remove the packages listed above, including datashader, and add all libcumlprims versions from nvidia to rapidsai

Testing package removals

Using a resurrected version of https://github.com/regro/conda-metachannel/ I was able to test the impact on various solves and environments by blocking packages currently available in the rapidsai channel (simulating the end-state if we move those packages under a label).

The current removal list consists of these files from rapidsai:

[linux-64] = ['blazingsql', 'blazingsql-build-env', 'blazingsql-notebook-env', 'clang', 'clx', 'cupy', 'faiss', 'faiss-gpu', 'faiss-proc', 'libclx', 'libcudf_cffi', 'libcypher-parser', 'libfaiss', 'libfaiss-avx2', 'libnvstrings', 'rapids-blazing', 'rapids-build-env', 'rapids-doc-env', 'rapids-notebook-env', 'rapids-scout', 'rapids-scout-local', 'strings_udf', 'ucx']

[linux-aarch64] = ['cupy', 'faiss', 'faiss-gpu', 'faiss-proc', 'libcypher-parser', 'libfaiss', 'nccl', 'pyarrow', 'rapids-build-env', 'rapids-notebook-env', 'rapids-scout-local', 'strings_udf', 'ucx']

[noarch] = ['cmake_setuptools', 'dask-xgboost', 'datashader', 'python-whois']

Results

All environments that could install were installed and then run against the test script linked above (https://gist.github.com/raydouglass/ff100a114c2a370b68131af55959afc0).

All x86 test runs were on my work laptop, running:

• Ubuntu 24.04
• Driver: 565.57.01
• RTX 3500 AD104GLM

All aarch64 test runs were on an NVIDIA labs machine, running:

• Ubuntu 22.04 LTS
• Driver: 535.161.08
• A100

The results are additionally divided by whether or not the single datashader tarball that is in rapidsai/noarch is removed or not. More on that below.

All of these installs are of the form:

mamba create -n {name} python={python_version} [cuda-version=12 | cudatoolkit=11.8] rapids={ver} -c rapidsai -c conda-forge -c nvidia (--strict-channel-priority)? (--platform=linux-aarch64)? --override-channels

With datashader=0.13.1a removed

RAPIDS version	arch	CUDA version	Installs	Strict Priority	Passes Tests	Failure reason
24.12	x86	12	Y	Y	Y
24.10	x86	12	Y	Y	Y
24.08	x86	12	Y	Y	Y
24.06	x86	12	Y	Y	Y
24.04	x86	12	Y	N	Y	missing libcumlprims
24.02	x86	12	Y	N	Y	missing libcumlprims
24.12	aarch64	12	Y	Y	Y
24.10	aarch64	12	Y	Y	Y
24.08	aarch64	12	Y	Y	Y
24.06	aarch64	12	Y	Y	Y
24.04	aarch64	12	Y	N	Y	missing libcumlprims
24.02	aarch64	12	Y	N	Y	missing libcumlprims

libcumlprims was added to rapidsai starting in 24.06, so strict solves fail before version 24.06.

RAPIDS version	arch	CUDA version	Installs	Strict Priority	Passes Tests	Failure reason
24.12	x86	11	Y	N	Y	cuda-profiler-api>=11.4.240,<12
24.10	x86	11	Y	N	Y	cuda-profiler-api>=11.4.240,<12
24.08	x86	11	Y	N	Y	cuda-profiler-api>=11.4.240,<12
24.06	x86	11	Y	N	Y	cuda-profiler-api>=11.4.240,<12
24.04	x86	11	Y	N	Y	libcumlprims
24.02	x86	11	Y	N	Y	libcumlprims
23.12	x86	11	Y	N	Y	libcumlprims
23.10	x86	11	Y	N	Y	libcumlprims
23.08	x86	11	Y	N	Y	libcumlprims
23.06	x86	11	Y	N	Y	libcumlprims
23.04	x86	11	Y	N	Y	libcumlprims
23.02	x86	11	N			datashader=0.13.1a
24.12	aarch64	11	Y	N	Y	cuda-profiler-api>=11.4.240,<12
24.10	aarch64	11	Y	N	Y	cuda-profiler-api>=11.4.240,<12
24.08	aarch64	11	Y	N	Y	cuda-profiler-api>=11.4.240,<12
24.06	aarch64	11	Y	N	Y	cuda-profiler-api>=11.4.240,<12
24.04	aarch64	11	Y	N	Y	libcumlprims
24.02	aarch64	11	Y	N	Y	libcumlprims
23.12	aarch64	11	Y	N	Y	libcumlprims
23.10	aarch64	11	Y	N	Y	libcumlprims
23.08	aarch64	11	Y	N	Y	libcumlprims
23.06	aarch64	11	Y	N	Y	libcumlprims
23.04	aarch64	11	Y	N	Y	libcumlprims
23.02	aarch64	11	N			datashader=0.13.1a

cuda-profiler-api is only available on conda-forge for >=12 so strict priority won’t work (cuda-profiler-api<12 can’t be added to conda-forge without also adding older versions of cuda-toolkit which seems like too much work for this effort.)

With datashader=0.13.1a not removed

If datashader=0.13.a is left in-place, then strict solves fail because conda is looking for datashader>=0.14 but the presence of datashader=0.13.a in rapidsai constrains the search space to the rapidsai channel only (for that particular dependency).

None of the rapids environments can install with --strict-channel-priority. But rapids-23.02 will install.

Recommendations

I would very much welcome opinions from the broader build team on this.

Below are a few general approaches and the top-level pros and cons, as I see them:

Remove the packages listed above, including datashader (recommended)

• All RAPIDS versions back to 23.04 install without changes to the install command.
• All RAPIDS versions >=24.06 with CUDA12 now solve with --strict-channel-priority.
• RAPIDS version 23.02 will need to specify a label in the install command for it to work.

Remove the packages listed above, don’t remove datashader and add all datashader versions >=0.14 to rapidsai

• All RAPIDS versions back to 23.04 install without changes to the install command.
• All RAPIDS versions >=24.06 with CUDA12 now solve with --strict-channel-priority.
• RAPIDS version 23.02 will install without changes to the install command.
• We have to build and upload several versions of datashader and continue doing so until we drop support for 23.02

Remove the packages listed above, including datashader, and add all libcumlprims versions from nvidia to rapidsai

• Outcome is the same as the recommended approach, but --strict-channel-priority might work for RAPIDS versions >=23.04 with CUDA12 (this is harder to test at the moment).
• We have to build and upload several versions of libcumlprims to rapidsai and continue doing so until we drop support for 24.04

[bdice]

Remove the packages listed above, including datashader (recommended)

Let's do this option. I think the outcome of "All RAPIDS versions back to 23.04 install without changes to the install command" meets our requirements for backwards-looking support. Strict channel priority is only important to us in a forward-looking context, so moving old versions of libcumlprims to rapidsai is not worthwhile in my opinion.

[raydouglass]

Wow thanks @gforsyth! I suspect supporting back about two years should be enough, but since there were skeptics, I've added clarifying this to the PIC sync agenda on Tuesday.

However, just want to clarify whether the tests were done with just linux-64 or linux-aarch64 (or both) architectures.

One concern I had is that we built and published several third-party linux-aarch64 packages, but I don't remember what RAPIDS versions. I suspect it was before 23.04, but it would be good to verify aarch64 works for the same versions as well.

[gforsyth]

However, just want to clarify whether the tests were done with just linux-64 or linux-aarch64 (or both) architectures.

Great point @raydouglass -- this was just on my work laptop, so linux-64 only. I'll look into options for testing against linux-aarch64

[raydouglass]

I'll look into options for testing against linux-aarch64

We have some arm64 machines in the RDS lab, you can file an issue to get access: https://github.com/rapidsai/ops/issues/new?template=02-rds-lab-machine-and-access-request.yml

[jakirkham]

One can also test other architectures on the same machine using the --platform flag

For example running this Windows install...

conda create --dry-run --platform win-64 --name tst_win_64 zlib

...gives me this on my Mac:

Details

Channels:
 - conda-forge
Platform: win-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/jkirkham/miniforge/envs/tst_win_64

  added / updated specs:
    - zlib


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libzlib-1.3.1              |       h2466b09_2          54 KB  conda-forge
    ucrt-10.0.22621.0          |       h57928b3_1         547 KB  conda-forge
    vc-14.3                    |      ha32ba9b_23          17 KB  conda-forge
    vc14_runtime-14.42.34433   |      he29a5d6_23         737 KB  conda-forge
    zlib-1.3.1                 |       h2466b09_2         105 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         1.4 MB

The following NEW packages will be INSTALLED:

  libzlib            conda-forge/win-64::libzlib-1.3.1-h2466b09_2 
  ucrt               conda-forge/win-64::ucrt-10.0.22621.0-h57928b3_1 
  vc                 conda-forge/win-64::vc-14.3-ha32ba9b_23 
  vc14_runtime       conda-forge/win-64::vc14_runtime-14.42.34433-he29a5d6_23 
  zlib               conda-forge/win-64::zlib-1.3.1-h2466b09_2 



DryRunExit: Dry run. Exiting.

[jakirkham]

Thanks for digging into this Gil! 🙏

Remove the packages listed above, including datashader (recommended)

Agree this is a good recommendation

I suspect supporting back about two years should be enough, but since there were skeptics, I've added clarifying this to the PIC sync agenda on Tuesday.

A tweak on Gil's approach could be to instead of deleting the packages to move them to a label, like legacy

For all practical purposes those packages would still be ignored for the solve. There would just be a way for users to get those back by adding this label

Another advantage of a label is we can always tweak it again if needed

Anyways agree this would be good to discuss. Thanks for bringing it up Ray! 🙏

[gforsyth]

Updated my results above with dry-run solves for linux-aarch64 and a few additional proposed "removals" (relabels).
Still need to test that those environments function as expected, but in terms of environment solutions, aarch64 matches x86 in terms of where strict solves will work and how far back things will continue to work.