Description
Series.sum(..., skipna=False) and DataFrame.sum(..., skipna=False) are very slow on a MultiIndex,
if compared to the same method called with (..., skipna=True)
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
(optional) I have confirmed this bug exists on the master branch of pandas.
Code Sample, a copy-pastable example
import pandas as pd
import numpy as np
dims = (1000, 100, 10)
# make random data
rg = np.random.default_rng(0)
x = rg.random(dims)
# construct dataframe with multindex
idx = pd.MultiIndex.from_product(map(range, dims), names=["i0", "i1", "i2"])
df = pd.Series(data=x.ravel(), index=idx, name="x")
# comupute sum over "i2" level in pandas
s1 = df.sum(level=["i0", "i1"], skipna=False)
s2 = df.sum(level=["i0", "i1"], skipna=True)
# compute sum over "axis=2" in numpy
s_ref = x.sum(axis=2)
# check results
s_ref_norm = np.linalg.norm(s_ref)
for s in (s1, s2):
print(np.linalg.norm(s.to_numpy().reshape(s_ref.shape) - s_ref) / s_ref_norm)Problem description
The computation of s1 = df.sum(level=["i0", "i1"], skipna=False) takes forever in the sample above:
In [2]: %timeit df.sum(level=["i0", "i1"], skipna=False)
8.44 s ± 47.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [3]: %timeit df.sum(level=["i0", "i1"], skipna=True)
52.1 ms ± 266 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [4]: %timeit x.sum(axis=2)
1.09 ms ± 13.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)Here we see a severe performance hit (8.61 s vs 0.05 s) for the very same DataFrame.sum if skipna=False.
Maybe this is related to #16788 and #37622
The performance of df.sum(level=["i0", "i1"], skipna=True) with respect to baseline numpy is not great, but I think this is not an issue here.
Results are correct, fortunately. Curiously the slow result (skipna=False) is identical with the reference numpy sum, while the faster result (skipna=True) is within ±2 ULP, but of course this is not a concern.
Almost the same results hold if we use a DataFrame instead of a Series
Output of pd.show_versions()
INSTALLED VERSIONS
commit : 67a3d42
python : 3.8.2.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8
pandas : 1.1.4
numpy : 1.19.4
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.4
setuptools : 49.6.0
Cython : None
pytest : 6.0.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.17.0
pandas_datareader: None
bs4 : None
bottleneck : 1.3.2
fsspec : 0.8.0
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : 1.0.1
pytables : None
pyxlsb : None
s3fs : 0.5.0
scipy : 1.5.3
sqlalchemy : None
tables : 3.6.1
tabulate : 0.8.7
xarray : 0.16.0
xlrd : 1.2.0
xlwt : None
numba : 0.51.1