⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

[scikit-learn-bot]

CI is still failing on Linux_Nightly.pylatest_pip_scipy_dev (Jul 06, 2023)
Unable to find junit file. Please see link for details.

[glemaitre]

/take

[glemaitre]

The culprit is pandas dev. I will bisect to know which commit changed the behaviour.

[glemaitre]

So it comes from this commit: pandas-dev/pandas#52542

It comes from calling pd.concat(..., ignore_index=True) with a first dataset containing None (thus an object dtype) with a second dataset containing np.nan and float (thus a float64 dtype).

The previous behaviour cast the column as object dtype while the new behaviour is casting into float64.

I am trying to craft a minimal reproducer.

[lesteve]

There are plenty of (~280) recent errors probably due to a pandas change (and maybe numpy too?), symptoms look like this:

• FutureWarning: is_sparse is deprecated and will be removed in a future version. Check isinstance(dtype, pd.SparseDtype) instead.
• FutureWarning: The behavior of DataFrame concatenation with all-NA entries is deprecated. In a future version, this will no longer exclude all-NA columns when determining the result dtypes. To retain the old behavior, cast the all-NA columns to the desired dtype before the concat operation.
• ValueError: setting an array element with a sequence.
• DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)

Number of failures look like this (quick and dirty analysis may miss a few kind of errors):

    245 FutureWarning: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
     19 DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
     14 ValueError: setting an array element with a sequence.
      5 FutureWarning: The behavior of DataFrame concatenation with all-NA entries is deprecated. In a future version, this will no longer exclude all-NA columns when determining the result dtypes. To retain the old behavior, cast the all-NA columns to the desired dtype before the concat operation.

[lesteve]

Opened #26287 about pandas is_sparse