[FEA] Migrate Python bindings to Cython for cp311 ABI3 builds #1151

Posted by codex GPT-5 on Michael Sarahan's behalf. Readers should treat this issue content as LLM-generated.

Summary

Migrate cuCIM's compiled Python bindings from pybind11 to Cython and build cucim against the CPython 3.11 Limited API, producing cp311-abi3 wheels and version-independent Conda packages.

Related: #1029. That issue identifies the correct underlying problem—pybind11 does not currently support the Limited API—and proposes nanobind. This issue keeps the Limited-API goal but proposes Cython as the implementation path.

Why this is needed

The Python dimension in the rapidsai/ci-imgs build-image matrix is expensive. A library that produces a cp311-abi3 extension only needs the minimum supported Python version for artifact construction; the resulting artifact can be installed and tested on later supported CPython versions.

The goal is therefore to make cuCIM buildable once per CUDA and architecture combination, without a separate build for every supported Python minor version. Runtime validation across Python 3.11 and later remains required; this proposal reduces the build matrix, not the test matrix.

Why Cython instead of nanobind

Python 3.11 support is a requirement.

Nanobind's directly linked stable-ABI mode requires Python 3.12 or later. Nanobind split mode can target an older Limited API, but it divides the extension into an ABI3 frontend and a separately distributed, CPython-version-specific backend. That model adds a new runtime binary dependency, requires frontend/backend C++ platform-ABI compatibility, and is not yet an established RAPIDS packaging pattern. It also makes ownership of the remaining Python-specific build matrix less intuitive.

Cython is preferred because:

• Cython's Limited-API path supports the required cp311-abi3 floor.
• cuDF, cuVS, and cuOpt already use Cython with rapids-cython-core, scikit-build-core, ABI3 wheels, version-independent Conda packages, stable artifact names, and build-once/test-many CI.
• It produces a self-contained cuCIM extension without a Python-specific runtime binding backend.
• It can retain cuCIM's existing _GLIBCXX_USE_CXX11_ABI=0 native-library contract. Moving that contract to the modern libstdc++ ABI would be a separate ABI-breaking decision and is not necessary for the Limited-API work.
• Using the same build and validation machinery as other RAPIDS repositories reduces long-term maintenance and makes failures easier to compare across projects.

Examples:

• cuDF wheel.py-api = "cp311"
• cuVS Cython initialization and cp311 wheel configuration
• cuOpt Cython initialization and cp311 wheel configuration

Proposed work

1. Capture binding compatibility before replacing pybind11

• Add or strengthen tests for the public behavior currently provided by cucim.clara._cucim.
• Cover CuImage, CuImageIterator, Device, ImageCache, Profiler, CuFileDriver, and exported enums.
• Verify constructor, context-manager, iterator, exception, and object-lifetime behavior.
• Verify metadata types and values.
• Verify __array_interface__ and __cuda_array_interface__, including attribute absence when an image has no backing data.
• Cover CPU buffers, CUDA array-interface inputs, readonly buffers, noncontiguous inputs, and filesystem pread/pwrite paths.
• Add deterministic ownership/leak coverage for repeated image construction, region reads, iteration, and destruction.

2. Implement Cython bindings

• Add .pxd declarations for the required public libcucim C++ types and functions.
• Implement Cython extension classes that own std::shared_ptr<T> explicitly, especially for CuImage, its iterator, caches, profilers, and file drivers.
• Preserve the existing public cucim and cucim.clara imports. Private compiled-module layout may change if Python compatibility shims retain the public API.
• Release the GIL around long-running native operations using Cython nogil declarations/blocks, while retaining the GIL for Python-object inspection and conversion.
• Implement metadata conversion without pybind11_json.
• Implement CPU and CUDA array-interface exposure with the same lifetime and ownership guarantees as the current bindings.
• Remove reliance on pybind11 internals and direct non-Limited-API macros such as PyTuple_SET_ITEM and PyMemoryView_GET_BUFFER.

3. Adopt the established RAPIDS Cython build pattern

• Use scikit-build-core through rapids-build-backend.
• Add Cython and rapids-cython-core as generated build dependencies.
• Replace pybind11_add_module() with rapids_cython_init() and rapids_cython_create_modules().
• Set wheel.py-api = "cp311".
• Continue linking against the existing cucim::cucim target and preserve _GLIBCXX_USE_CXX11_ABI=0.
• Remove the vendored/patched pybind11 and pybind11_json dependencies after the Cython extension fully replaces them.
• Remove the setuptools workaround that marks a prebuilt, Python-specific shared object as an extension.

4. Make wheel and Conda outputs ABI3/version-independent

• Produce extension filenames ending in .abi3.so.
• Produce wheels tagged cp311-abi3.
• Mark the Conda Python package version-independent (python_version_independent: true in the current recipe format, or the equivalent if the recipe is migrated).
• Build the Conda package against Python 3.11 and python-abi3 3.11.*.
• Use a cp311_abi3 Conda build string.
• Use stable RAPIDS artifact names that do not include the runtime Python version.

5. Validate artifacts before reducing the matrix

• Run abi3audit --strict --summary --verbose on built wheels.
• Inspect wheel contents and metadata to confirm the cp311-abi3 tag and .abi3.so extension.
• Inspect the Conda artifact for .abi3.so, _python_abi3_support, and an appropriate cpython >=3.11 runtime constraint.
• Build each wheel and Conda package once with Python 3.11.
• Install and test those exact artifacts on every supported CPython version, currently Python 3.11 through 3.14.
• Run the full Python tests, binding-focused ownership tests, and relevant CPU/GPU memory tests on those artifacts.

6. Reduce the build matrix and update ci-imgs

• After artifact validation passes, filter cuCIM wheel and Conda build jobs to the minimum supported Python version for each CUDA and architecture combination. Follow the established form:

  group_by({CUDA_VER, ARCH})
  | map(min_by(.PY_VER | split(".") | map(tonumber)))

• Keep Python 3.11-3.14 runtime test coverage.

• Update artifact download consumers to use the stable artifact name.

• Follow up in rapidsai/ci-imgs to remove build-image variants that existed only to compile cuCIM for each Python minor version.

• Measure and record the resulting image-build and cuCIM build-job reduction.

Suggested sequencing

1. Characterization tests and the Cython binding conversion, initially retaining the existing build matrix.
2. ABI3 wheel/Conda configuration and build-once/test-many artifact proof.
3. Stable artifact naming, cuCIM build-matrix reduction, and the corresponding ci-imgs reduction.

Keeping these stages separate makes binding regressions distinguishable from packaging, solver, artifact-routing, or matrix-filter failures.

Acceptance criteria

• The supported public Python API and observable array-interface behavior remain compatible.
• The Cython extension is built against the CPython 3.11 Limited API.
• A wheel built once on Python 3.11 installs and passes tests unchanged on Python 3.11-3.14.
• A version-independent Conda artifact built once installs and passes tests unchanged on Python 3.11-3.14.
• abi3audit passes in strict mode.
• Artifact inspection confirms .abi3.so, cp311-abi3, and correct Conda ABI3 metadata.
• Wheel and Conda build jobs no longer vary by Python minor version.
• The runtime test matrix continues to cover every supported Python minor version.
• No change to the libcucim C++ ABI is required as part of this work.

Non-goals

• Dropping Python 3.11 support.
• Reducing supported-Python runtime test coverage.
• Changing the public cuCIM Python API.
• Changing libcucim's libstdc++ ABI contract.
• Introducing a Python-version-specific runtime binding backend.