issubdtype(<categorical>, np.bool_) raises error

[jankatins]

In[14]: import pandas as pd
Backend Qt4Agg is interactive backend. Turning interactive mode on.
In[15]: import numpy as np
In[16]: s = pd.Series([1,2,3,1,2,3]).astype("category")
In[17]: s
Out[17]: 
0    1
1    2
2    3
3    1
4    2
5    3
dtype: category
Categories (3, int64): [1 < 2 < 3]
In[18]: np.issubdtype(s.dtype, np.bool_)
Traceback (most recent call last):
  File "c:\data\external\ipython\IPython\core\interactiveshell.py", line 3032, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-18-607a91e2a828>", line 1, in <module>
    np.issubdtype(s.dtype, np.bool_)
  File "C:\portabel\miniconda\envs\ipython\lib\site-packages\numpy\core\numerictypes.py", line 763, in issubdtype
    return issubclass(dtype(arg1).type, arg2)
TypeError: data type not understood

This is a problem in pydata/patsy#47

Not sure if there is an easy way to get numpy to understand this (I've absolutely no numpy foo :-/ ). If not this means that every patsy/statsmodels method which does dtype magic has to guard against the category dtype :-/

[jreback]

numpy doesn't understand this

you could do

isinstance(s.dtype, np.dtype) and np.issubdtype(s.dtype, np.bool_)

[jankatins]

Just understood that this is again the dtype("category") problem :-(

[jreback]

yep, you can also do com.is_categorical_dtype(....) which is safe.

[jankatins]

The problem is more that every other dtype check is prone to raise :-(

This is the line which raises (https://github.com/pydata/patsy/blob/master/patsy/categorical.py#L185)

# fastpath to avoid doing an item-by-item iteration over boolean
        # arrays, as requested by #44
        if hasattr(data, "dtype") and np.issubdtype(data.dtype, np.bool_): # <= !!! this line !!!
            self._level_set = set([True, False])
            return True

-> if you want to work with pandas categorical data, you now have to look out for any (arbitrary) method, which errors out when getting a non-official dtype :-/

[jankatins]

IMO the most user friendly solution would be to monkey-patch this numpy function to check for categorical first...

[jreback]

you need to dispatch on categorical before this and treat it separately. Code that deals with multiple dtypes needs to be aware. You can simply np.asarray if you want on everything.

You cannot patch a c-level function.

[jankatins]

np.issubdtype is not a c level function :-)

https://github.com/numpy/numpy/blob/master/numpy/core/numerictypes.py#L736

def issubdtype(arg1, arg2):
    if issubclass_(arg2, generic):
        return issubclass(dtype(arg1).type, arg2)
    mro = dtype(arg2).type.mro()
    if len(mro) > 1:
        val = mro[1]
    else:
        val = mro[0]
    return issubclass(dtype(arg1).type, val)

[jankatins]

The problem is that (this and probably other) code worked great before pandas 0.15 and works great as long as no categorical comes along but then breaks if you pass in a categorical. So more or less the categorical introduction breaks unrelated code. :-(

[jreback]

not sure what to tell you
the numpy functions are just not safe
you might have better luck doing
np.asarray(...) then checking that dtype

[jankatins]

So, push it to numpy-land?

[jorisvandenbossche]

Couldn't this be solved by returning the CategoricalDType object instead of a string for s.dtype?

In [86]: catdt = pd.core.categorical.CategoricalDtype

In [87]: np.issubdtype(catdt, np.bool_)
Out[87]: False

EDIT: I mean an instance of the object instead of string -> so returning the class instead of an instance for s.dtype

[jreback]

@JanSchulz I would explicity check or a Categorical in patsy. You need to handle them specially anyhow.

[jreback]

@jorisvandenbossche
Its already a dtype, just not in the numpy hierarchy (and they are not careful about the check)

In [1]: s = Series(list('abc')).astype('category')

In [2]: s.dtype
Out[2]: category

In [3]: type(s.dtype)
Out[3]: pandas.core.common.CategoricalDtype

[jankatins]

I'm confused:

In[25]: type(s.dtype)
Out[25]: pandas.core.common.CategoricalDtype
In[26]: catdt = pd.core.categorical.CategoricalDtype
In[27]:  np.issubdtype(catdt, np.bool_)
Out[27]: False
In[28]: np.issubdtype(s.dtype, np.bool_)
Traceback (most recent call last):
  File "c:\data\external\ipython\IPython\core\interactiveshell.py", line 3032, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-28-607a91e2a828>", line 1, in <module>
    np.issubdtype(s.dtype, np.bool_)
  File "C:\portabel\miniconda\envs\ipython\lib\site-packages\numpy\core\numerictypes.py", line 763, in issubdtype
    return issubclass(dtype(arg1).type, arg2)
TypeError: data type not understood
In[29]: type(s.dtype) is type(catdt)
Out[29]: False
In[30]: type(s.dtype), type(catdt)
Out[30]: (pandas.core.common.CategoricalDtype, type)
In[31]: type(np.bool_)
Out[31]: type