[python-package] NumPy 2.0 support

[jameslamb]

Summary

numpy 2.0 is coming out soon, and has release candidates up.

We should test lightgbm against it, and if any changes are required, try to get them into the next release (#6439).

Motivation

numpy is a hard runtime dependency of lightgbm. It would not be a good experience for users if incompatibility with its new major version leads to needing to manually pin numpy<2 in their environments.

Description

At a minimum, we should manually test lightgbm against the numpy release candidate. See dmlc/treelite#560 for a migration guide.

It also would be great to set up CI that runs against nightlies of numpy, pandas, scikit-learn, and scipy, maybe following xarray's example: https://github.com/pydata/xarray/blob/31111b3afe44fd6f7dac363264e94186cc5168d2/.github/workflows/upstream-dev-ci.yaml

References

Links to other projects' upgrade and testing efforts: numpy/numpy#26191

[jameslamb]

There will be at least one thing I know needs to change... np.array(..., copy=False) calls being changed to np.asarray(...). Reference: https://github.com/dmlc/treelite/pull/562/files#r1603610473