feat: ibis_ml.metrics

[IndexSeek]

Explore adding a metrics module to support computing classification metrics such as accuracy, precision, recall, and f1_score, as well as regression metrics such as MSE, RMSE, MAE, and MAPE.

This was brought up in ibis-project/ibis#10501.

A challenge with this implementation, at least for classification, is that calculating these metrics individually may be less performant, as it likely requires reshaping the data first for each one. Even with this being the case, we would likely still get pretty good performance, and if the backend supports result caching, it would be even better in those instances.

[zilto]

To add: torchmetrics.functional has a lot of metrics implemented for PyTorch tensors in functional form. The typical input is metric(predicted, true, **kwargs), which maps to table columns.

A challenge with this implementation, at least for classification, is that calculating these metrics individually may be less performant, as it likely requires reshaping the data first for each one

Some metrics are implemented modularly with an _update() and a _compute() function. update derives relevant values from the predicted and true inputs and compute produces the metric. (example)

Notes

• ibis-ml.metrics implementations could be tested against torchmetrics (or scikit-learn) itself
• some metrics can be "accumulated", which means that we can update the metric value for a single new pair (predicted, true)

[deepyaman]

To add: torchmetrics.functional has a lot of metrics implemented for PyTorch tensors in functional form.

@zilto Sorry I didn't get back to you on this! If any of these would be useful for your work, or if you're more generally interested in contributing them, more than happy to add to the module @IndexSeek created! I'd also of course love to see some use in the wild—validating that this is a real challenge a number of people are facing—before going and implementing all of the metrics. :D

• ibis-ml.metrics implementations could be tested against torchmetrics (or scikit-learn) itself

Yep! @IndexSeek's initial set of implementations are all tested against their scikit-learn equivalents, and I find it quite clean.