Skip to content

DataArray.chunk() from sparse array produces malformed dask array #3191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
crusaderky opened this issue Aug 8, 2019 · 1 comment · Fixed by #3202
Closed

DataArray.chunk() from sparse array produces malformed dask array #3191

crusaderky opened this issue Aug 8, 2019 · 1 comment · Fixed by #3202

Comments

@crusaderky
Copy link
Contributor

crusaderky commented Aug 8, 2019

#3117 by @nvictus introduces support for sparse in plain xarray.
dask already supports it.

Running with:

  • xarray git head
  • dask 2.2.0
  • numpy 1.16.4
  • sparse 0.7.0
  • NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1
>>> import numpy, sparse, xarray, dask.array
>>> s = sparse.COO(numpy.array([1, 2]))         
>>> da1 =  dask.array.from_array(s)
>>> da1._meta
<COO: shape=(0,), dtype=int64, nnz=0, fill_value=0>
>>> da1.compute()
<COO: shape=(2,), dtype=int64, nnz=2, fill_value=0>
>>> da2 = xarray.DataArray(s).chunk().data
>>> da2._meta                                                                                                                                                                                                      
array([], dtype=int64)  # Wrong
>>> da2.compute()
RuntimeError: Cannot convert a sparse array to dense automatically. To manually densify, use the todense method.
@shoyer
Copy link
Member

shoyer commented Aug 10, 2019

I'm pretty sure this is what breaks things:

# For now, assume that all arrays that we wrap with dask (including
# our lazily loaded backend array classes) should use NumPy array
# operations.
if LooseVersion(dask.__version__) > "1.2.2":
kwargs = dict(meta=np.ndarray)
else:
kwargs = dict()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants