Extension integer dtypes (such as pd.Int32Dtype) do not work with numexpr

[techvslife]

In the latest pandas version (0.24.1), combined with the latest version of numexpr (2.6.9), extension dtypes do not work (e.g. the “query” method on dataframes fails when the engine parameter is set to the default ):

https://stackoverflow.com/questions/54759936/extension-dtypes-in-pandas-appear-to-have-a-bug-with-query

Code to reproduce:
df_test = pd.DataFrame(data=[4,5,6], columns=["col_test"])
df_test = df_test.astype(dtype={"col_test": pd.Int32Dtype()})
df_test.query("col_test != 6")
Last lines of the long error message are:

File "...\site_packages\numexpr\necompiler.py", line 822, in evaluate zip(names, arguments)] File "...\site_packages\numexpr\necompiler.py", line 821, in signature = [(name, getType(arg)) for (name, arg) in File "...\site_packages\numexpr\necompiler.py", line 703, in getType raise ValueError("unknown type %s" % a.dtype.name) ValueError: unknown type object

Thanks!

[robbmcleod]

Yeah, pandas has this problem with not discussing what they are doing in any sort of open forum. Like I'm subscribed to the [Pandas-dev] mailing list but I never hear what they are actually doing, so this is a complete surprise that they have changed the numpy dtype protocol. We are upstream of pandas, and numpy is upstream of us, so if pandas wants to change how numpy dtypes work, they have to discuss that with [Numpy-discussion].

[techvslife]

Thank you. My understanding is that this is independent of numpy dtypes—they are pandas dtypes (which they call extended dtypes) to allow for such things as nullable integers. I am not sure of the extent to which these types are meant to integrate into numpy, or how they do. I’ll put a post on the numpy github, and also play around in numpy and see if there is any compatibility with the new pandas dtypes. Here is the pandas report I posted: pandas-dev/pandas#25369

[techvslife]

just fyi: the main documentation reference is here: https://pandas.pydata.org/pandas-docs/stable/development/extending.html That mentions that all pandas extension types must be convertible to a numpy array (even if it’s an expensive operation). Here is the dtype I was using, a pandas nullable integer (pd.Int32Dtype—capitalized, instead of np.int32): https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html

[robbmcleod]

Sounds like pandas should be raising an exception here.

[robbmcleod]

Do we have an issue raised with pandas for this issue? In my opinion pandas should be erroring before this falls down into numexpr since it's non-standard with regards to NumPy. I can certainly raise this issue with them if they aren't doing that.

[techvslife]

I have a post with the pandas guys at:
pandas-dev/pandas#25369
and with the numpy guys at:
numpy/numpy#13330

I believe the pandas team consider the issue an enhancement request, rather than a bug, so while they're open to someone fixing it as you suggest, it's not a priority. Hopefully as more people use the new nullable integer types, the issue will get assigned to someone for a fix.

[robbmcleod]

Closing as pandas has closed the issue on their side.

[techvslife]

I see the issue marked as “open” in pandas:
pandas-dev/pandas#25369

Or am I looking in the wrong place?