Replace conda-build with rattler-build

RAPIDS currently builds conda packages in CI using conda-build. The rattler-build tool is a newer alternative. It is written in Rust, and should be faster than conda-build (I haven't seen any official benchmarks yet, though). It only supports a limited subset of the meta.yaml recipe format, but that subset is designed to still enable all the same features, just with a more limited syntax (see CEPS 13 and 14). conda-build overhead is nontrivial (I've never benchmarked it, but I know it can stretch into multiple minutes beyond the environment solve when doing local CI reproductions), and reducing that would be quite valuable for us in improving our CI turnaround. Moreover, switching to the more restricted syntax described in the above CEPs would be beneficial because it would convert our conda recipes into pure YAML rather than the extended YAML currently used by meta.yaml. That change is important because the YAML extensions currently in our recipe make it impossible to parse or write with standard YAML parsers, which is a big reason why we have struggled to do things like support meta.yaml files in rapids-dependency-file-generator.

We should do a PoC of replacing conda-build with rattler-build in one repo (preferably something reasonably complex like cudf or cugraph) to see what it would take to make this transition, and how much we would benefit.

`rattler-build` porting progress

Few more:

nx-cugraph feat(rattler): port conda build recipe to rattler build nx-cugraph#121
cugraph-gnn feat(rattler): port conda build recipe to rattler build cugraph-gnn#178
cuopt https://github.com/rapidsai/cuopt/pull/2420

Follow-up work

I'm going to track a few rattler-build followups here and update as they get resolved:

general

Look into rolling the sccache cache-busting fix for -fdebug-prefix-map into rapids-cmake (xref rapidsai/rapids-cmake#798 (comment))

cudf

Add default build arg generation to rapids-rattler-channel-string #156
Follow up and remove extra wrapping quotes in context once Values of context variables can have different types depending on the underlying shell prefix-dev/rattler-build#1451 is resolved
Follow up and remove sed workaround for -fdebug-prefix-map once Cache build including output version in environment variables is causing sccache misses prefix-dev/rattler-build#1458 is resolved
Remove cuda-nvcc-impl dependency once it is provided by numba-cuda (xref [BUG] Add proper dependencies to conda package NVIDIA/numba-cuda#146)
Strip out secrets and env from cudf-polars recipe
Test removing cuda-version from all pure Python packages (xref Properly support building pure Python packages #43)
Confirm rapids-dask-dependency isn't needed by custreamz and then remove it
Figure out if python-confluent-kafka should have run exports
Confirm streamz is only a test dependency of custreamz and then remove it

rmm

Add default build arg generation to rapids-rattler-channel-string #156
Follow up and remove extra wrapping quotes in context once Values of context variables can have different types depending on the underlying shell prefix-dev/rattler-build#1451 is resolved

cuvs

Relax/extend pin mkl=2023 to mkl>=2023 or similar
combine cuvs-bench and cuvs-bench-cpu recipes

kvikio

The libcufile dependencies are an absolute mess and we would probably benefit from adding a few variants at the intersection of architecture and cuda version Port all conda recipes to rattler-build kvikio#678 (comment)

I now have two POCs of using rattler-build

Some notes:

rattler-build requires strict channel priority (Allow to change channel priority prefix-dev/rattler-build#343) so we need to address Switch to using strict channel priority during RAPIDS builds #84 first
Our general model of "build-once, install into separate outputs" requires a feature that rattler-build still marks as experimental, the multi-output cache. This doesn't personally concern me all that much since they are developing quite fast and I expect bug fixes to happen far more quickly than with conda-build, but it is something to be aware of.
There are some issues around cache run exports that would be best to have addressed upstream before we adopt.
- Update: This is now fixed in the latest release (0.19).

I've littered the PRs liberally with comments, so those should provide additional information.

Nice! Looks like you got some pretty good speedup there - maybe around 5 minutes of speedup, from what I can see. Considering the whole build time was 8-10 minutes, saving 5 minutes is huge.

Yup, similar speedups in absolute numbers for cugraph (smaller percentage, but still not bad to go from 28 mins to 22 mins. Also there's probably an extra 30 s in there for installing rattler-build itself in each job since it's not yet preinstalled in the image (but it will be).

I looked at the two POCs for ucxx and cugraph and I am supportive of this effort. It seems like there are a lot of steps to take first (like fixing strict priority conda builds, maybe altering librmm run exports, and maybe some rattler-build issues -- basically all the things that are marked in the comments) but overall I think this is mature enough that we can invest some time in this direction.

@wolfv

FYI @wolfv @ruben-arts here's the follow-up to our discussions from SciPy 🙂

We're actively discussing #84, which is IMO a blocker to making this change. We can work around it, but it requires a lot of extra boilerplate in recipes that I don't think it's worth proliferating.

Adding repo tracker for progress tracking:

rmm (Port to rattler-build rmm#1796)
raft
cudf (Port all conda recipes to rattler-build NVIDIA/cudf#18054)
kvikio
cuml
cucim
cugraph (Replace conda-build with rattler-build cugraph#4551)
cugraph-ops
cumlprims_mg
cuspatial
cuvs
cuxfilter
dask-cuda
rapids-cmake
rapids-dask-dependency
ucx-py
ucxx(Build conda packages using rattler-build ucxx#248)
wholegraph

@msarahan

We should re-evaluate this now that prefix-dev/rattler-build#343 is closed. I am not sure if strict channel priority is still a blocker for us to migrate to rattler-build. We need to see if prefix-dev/rattler-build#1211 solved the problems we were seeing.

cc: @msarahan @gforsyth @vyasr @jakirkham

prefix-dev/rattler-build#343 is indeed the issue that was blocking us, and it looks like it was resolved about a month ago which is great news. It should allow us to proceed on both fronts in parallel. I think enabling strict channel priority is worthwhile enough that we may as well move forward with that as well now given that we have enough information about how we can do it and approval to do so. It should give additional improvements to both user experiences and our solve times.

I want to tack on a task here while we refactor our conda recipes: remove the + environ.get('VERSION_SUFFIX', '') from all our package version strings. We never use a VERSION_SUFFIX, it is leftover from when we used versioneer long ago. See examples: https://github.com/search?q=org%3Arapidsai+%2F%28%3F-i%29%5C%2B+environ.get%5C%28%27VERSION_SUFFIX%27%2C+%27%27%5C%29%2F&type=code

We had this as a task item 2 years ago when we switched to GitHub Actions and we never got it cleaned up. Here is a list of related task items, which we should re-evaluate as we rewrite the conda recipes. Some may be irrelevant or already completed now. https://github.com/rapidsai/ops/issues/2447#issuecomment-1343422135

@bdice

The items originally listed in https://github.com/rapidsai/ops/issues/2447#issuecomment-1343422135 were later migrated to https://github.com/rapidsai/ops/issues/2537, which is the more up-to-date list. Most of the remaining unchecked items probably just require a single pass through the repos to see where there might be lingering issues because I think we've largely addressed them:

Change meta.yaml source.git_url to source.path to allow for local builds
Removing extra sccache env vars (at least cuxfilter is done, but we might find other places where we could improve this still)
Replace environ.get('XXX', 'YY') w/ just environ['XXX'] for env vars RAPIDS_CUDA_VERSION and CONDA_PY (@bdice just made a couple of PRs to help this along, and I suspect a quick pass through can get the rest)
Ensure rapids-*-env packages don't exist in dependencies.yaml (I believe this is done now)
Update version pinnings for libcu* so that we use <NEXT instead of <=HIGHEST to ensure compatibility with potential releases of CUDA 11.8.X while keeping below the known CUDA 12 version numbers (this just seems outdated given that our modern cuda-version pinning strategies for compatibility take this into account)

I'm not sure the following are worthwhile:

junit output annotations (not sure what it buys us)
consolidating CBCs (I think we have generally moved away from centralizing versions towards instead performing synchronized updates with something like rapids-reviser; if we do decide to centralize, I would prefer that we handle this by getting rapids-dependency-file-generator to support recipe.yaml files, which should be feasible after the rattler-build migration, and then implementing a centralized template store in dfg)

Last, this item is a minor quality of life improvement that we might as well do during the rattler migration but isn't really important enough to block anything on.

For packages with multiple mambabuild invocations (typically multiple Python packages), separate the outputs with rapids-logger notices like "Building cudf", "Building dask-cudf", etc. Applies to cudf, raft, cugraph, others?

Port to rattler-build (#1796)