[FEA] Model precision with precision_score

I'd like to be able to use cuML to calculate the precision score of an estimator in Python. I didn't see a C++/CUDA implementation in the metrics/ header files, so this may require both C++/CUDA and Python/Cython bindings eventually. Didn't see an existing issue but will close if I find one.

Currently, this requires going to the CPU if we want to use an existing API (relying on NEP-18 for dispatch doesn't work here). With about three million rows, sklearn's precision_score can take about 1.5 seconds. A stopgap GPU implementation for binary precision using CuPy can be 500x+ faster.

def cupy_precision_score(y, y_pred):
    
    pos_pred_ix = cupy.where(y_pred == 1)
    tp_sum = (y_pred[pos_pred_ix] == y[pos_pred_ix]).sum()
    fp_sum = (y_pred[pos_pred_ix] != y[pos_pred_ix]).sum()

    return (tp_sum / (tp_sum + fp_sum)).item()

%time print(cupy_precision_score(y, yp.values))
%time print(precision_score(y_host, y_pred))
0.9998934583422118
CPU times: user 2.59 ms, sys: 0 ns, total: 2.59 ms
Wall time: 2.05 ms
0.9998934583422118
CPU times: user 1.47 s, sys: 59.5 ms, total: 1.53 s
Wall time: 1.5 s

Cross-listing to the tracking issue #608

@beckernick

@beckernick starting with a CuPy implementation in cuML is totally fine. Particularly if submitted by the awesome person that is requesting it ;)

@cjnolet

@cjnolet This is the n-class version of the loose code above. It doesn't handle all of what sklearn does, of course. But it gets the core averages (binary, none, and macro). 100-300x speedup on low millions of rows.

Someone could perhaps run with this for an initial CuPy version as a good first issue.

import cupy as cp
import numpy as np

def cupy_precision_score(y, y_pred, average='binary'):
    """
    TODO: Handle the following
        - average=micro (slightly more annoying)
        - average=weighted (slightly more annoying)
    """
    nclasses = len(cp.unique(y))
    
    if average == 'binary' and nclasses > 2:
        raise ValueError
        
    if nclasses < 2:
        raise ValueError("Single class precision is not yet supported")
        
    res = cp.zeros(nclasses)
    
    for i in range(nclasses):
        pos_pred_ix = cp.where(y_pred == i)[0]
        
        # short circuit
        if len(pos_pred_ix) == 0:
            res[i] = 0
            break
            
        tp_sum = (y_pred[pos_pred_ix] == y[pos_pred_ix]).sum()
        fp_sum = (y_pred[pos_pred_ix] != y[pos_pred_ix]).sum()
        res[i] = (tp_sum / (tp_sum + fp_sum)).item()
        
    if not average:
        return res.get()
    elif average == 'binary':
        return res[nclasses-1].item()
    elif average == 'macro':
        return res.mean().item()
    return res.get()

Related to #242 . May want to separate the scorers into individual issues and keep using #242 as a tracker.

Hi, I'd like to pick this issue up if there's no one else working on it (I'm currently an intern working at NVIDIA).

@JohnZed

cc @JohnZed

@beckernick

Hey! I'm a first-time contributor and would love to pick this up using the approach suggested by @beckernick. Is the issue still open for contribution?

@dantegd

Hi, I'd like to pick this up if it is still free. Linked PR #3184 has been dormant since late 2021 and the last activity here was months ago, so checking first that nobody is actively working on it.

Plan, following @dantegd's CuPy guidance above and @beckernick's sketch:

Pure-CuPy precision_score in cuml.metrics, implemented next to accuracy_score using the current input-validation helpers
sklearn-compatible signature: average in {None, 'binary', 'macro', 'micro', 'weighted'} plus labels, pos_label and zero_division, computed from the GPU confusion matrix rather than a per-class loop
Tests against sklearn across numpy/cupy/cudf/pandas inputs, following the conventions in tests/test_metrics.py

If @ptartan21 or @Nzouh are still on this, happy to stand down.

Opened PR #8524 implementing this with the CuPy approach described above.

Navigation Menu

Repository navigation

[FEA] Model precision with precision_score #1522

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[FEA] Model precision with precision_score #1522

Description

Activity

dantegd commented on Dec 30, 2019

beckernick commented on Feb 12, 2020

beckernick commented on Feb 12, 2020

ptartan21 commented on Nov 10, 2020

beckernick commented on Nov 11, 2020

Nzouh commented on Mar 29, 2026

VaggelisGian commented on Aug 26, 2026

VaggelisGian commented on Aug 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions