Merge branch 'main' into median-aggregate

Author: crusaderky

.github/workflows/test.yaml

@@ -25,25 +25,39 @@ jobs:
       fail-fast: false
       matrix:
         python-version: ["3.9", "3.10", "3.11", "3.12"]
-        environment-file: [ci/environment.yml, ci/environment_released.yml]
+        environment-file: [ci/environment.yml]
 
     steps:
       - uses: actions/checkout@v2
         with:
           fetch-depth: 0  # Needed by codecov.io
 
+      - name: Get current date
+        id: date
+        run: echo "date=$(date +%Y-%m-%d)" >> "${GITHUB_OUTPUT}"
+
       - name: Install Environment
         uses: mamba-org/setup-micromamba@v1
         with:
           environment-file: ${{ matrix.environment-file }}
           create-args: python=${{ matrix.python-version }}
-          cache-environment: true
+          # Wipe cache every 24 hours or whenever environment.yml changes. This means it
+          # may take up to a day before changes to unpinned packages are picked up.
+          # To force a cache refresh, change the hardcoded numerical suffix below.
+          cache-environment-key: environment-${{ steps.date.outputs.date }}-0
 
       - name: Install dask-expr
         run: python -m pip install -e . --no-deps
 
+      - name: Print dask versions
+        # Output of `micromamba list` is buggy for pip-installed packages
+        run: pip list | grep -E 'dask|distributed'
+
       - name: Run tests
         run: py.test -n auto --verbose --cov=dask_expr --cov-report=xml
 
+      - name: Run Dask DataFrame tests
+        run: python -c "import dask.dataframe as dd; dd.test_dataframe()"
+
       - name: Coverage
         uses: codecov/codecov-action@v3

README.md

@@ -62,349 +62,7 @@ production settings.
 API Coverage
 ------------
 
-**`dask_expr.DataFrame`**
+Dask-Expr covers almost everything of the Dask DataFrame API. The only missing features are:
 
-- `abs`
-- `add`
-- `add_prefix`
-- `add_sufix`
-- `align`
-- `all`
-- `any`
-- `apply`
-- `assign`
-- `astype`
-- `bfill`
-- `clip`
-- `combine_first`
-- `copy`
-- `count`
-- `cummax`
-- `cummin`
-- `cumprod`
-- `cumsum`
-- `dask`
-- `div`
-- `divide`
-- `drop`
-- `drop_duplicates`
-- `dropna`
-- `dtypes`
-- `eval`
-- `explode`
-- `ffill`
-- `fillna`
-- `floordiv`
-- `groupby`
-- `head`
-- `idxmax`
-- `idxmin`
-- `ìloc`
-- `index`
-- `isin`
-- `isna`
-- `join`
-- `map`
-- `map_overlap`
-- `map_partitions`
-- `mask`
-- `max`
-- `mean`
-- `memory_usage`
-- `memory_usage_per_partition`
-- `merge`
-- `min`
-- `min`
-- `mod`
-- `mode`
-- `mul`
-- `nlargest`
-- `nsmallest`
-- `nunique_approx`
-- `partitions`
-- `pivot_table`
-- `pow`
-- `prod`
-- `query`
-- `radd`
-- `rdiv`
-- `rename`
-- `rename_axis`
-- `repartition`
-- `replace`
-- `reset_index`
-- `rfloordiv`
-- `rmod`
-- `rmul`
-- `round`
-- `rpow`
-- `rsub`
-- `rtruediv`
-- `sample`
-- `select_dtypes`
-- `set_index`
-- `shift`
-- `shuffle`
-- `sort_values`
-- `std`
-- `sub`
-- `sum`
-- `tail`
-- `to_parquet`
-- `to_timestamp`
-- `truediv`
-- `var`
-- `visualize`
-- `where`
-
-
-**`dask_expr.Series`**
-
-- `abs`
-- `add`
-- `align`
-- `all`
-- `any`
-- `apply`
-- `astype`
-- `between`
-- `bfill`
-- `clip`
-- `combine_first`
-- `copy`
-- `count`
-- `cummax`
-- `cummin`
-- `cumprod`
-- `cumsum`
-- `dask`
-- `div`
-- `divide`
-- `drop_duplicates`
-- `dropna`
-- `dtype`
-- `explode`
-- `ffill`
-- `fillna`
-- `floordiv`
-- `groupby`
-- `head`
-- `idxmax`
-- `idxmin`
-- `index`
-- `isin`
-- `isna`
-- `map`
-- `map_partitions`
-- `mask`
-- `max`
-- `mean`
-- `memory_usage`
-- `memory_usage_per_partition`
-- `min`
-- `min`
-- `mod`
-- `mode`
-- `mul`
-- `nlargest`
-- `nsmallest`
-- `nunique_approx`
-- `partitions`
-- `pow`
-- `prod`
-- `product`
-- `radd`
-- `rdiv`
-- `rename`
-- `rename_axis`
-- `repartition`
-- `replace`
-- `reset_index`
-- `rfloordiv`
-- `rmod`
-- `rmul`
-- `round`
-- `rpow`
-- `rsub`
-- `rtruediv`
-- `shift`
-- `shuffle`
-- `std`
-- `sub`
-- `sum`
-- `tail`
-- `to_frame`
-- `to_timestamp`
-- `truediv`
-- `unique`
-- `value_counts`
-- `var`
-- `visualize`
-- `where`
-
-
-**`dask_expr.Index`**
-
-- `abs`
-- `align`
-- `all`
-- `any`
-- `apply`
-- `astype`
-- `clip`
-- `combine_first`
-- `copy`
-- `count`
-- `dask`
-- `dtype`
-- `fillna`
-- `groupby`
-- `head`
-- `idxmax`
-- `idxmin`
-- `index`
-- `isin`
-- `isna`
-- `map_partitions`
-- `max`
-- `memory_usage`
-- `min`
-- `min`
-- `mode`
-- `nunique_approx`
-- `partitions`
-- `prod`
-- `rename`
-- `rename_axis`
-- `repartition`
-- `replace`
-- `reset_index`
-- `round`
-- `shuffle`
-- `std`
-- `sum`
-- `tail`
-- `to_frame`
-- `to_timestamp`
-- `var`
-- `visualize`
-
-
-**`dask_expr._groupby.GroupBy`**
-
-- `agg`
-- `aggregate`
-- `apply`
-- `bfill
-- `count`
-- `ffill`
-- `first`
-- `last`
-- `max`
-- `mean`
-- `median`
-- `min`
-- `nunique`
-- `prod`
-- `shift`
-- `size`
-- `std`
-- `sum`
-- `transform`
-- `value_counts`
-- `var`
-
-Support for ``SeriesGroupBy`` and ``DataFrameGroupBy``.
-
-**`dask_expr._resample.Resampler`**
-
-- `agg`
-- `count`
-- `first`
-- `last`
-- `max`
-- `mean`
-- `median`
-- `min`
-- `nunique`
-- `ohlc`
-- `prod`
-- `quantile`
-- `sem`
-- `size`
-- `std`
-- `sum`
-- `var`
-
-
-**`dask_expr._rolling.Rolling`**
-
-- `agg`
-- `apply`
-- `count`
-- `max`
-- `mean`
-- `median`
-- `min`
-- `quantile`
-- `std`
-- `sum`
-- `var`
-- `skew`
-- `kurt`
-
-
-**Binary operators (`DataFrame`, `Series`, and `Index`)**:
-
-- `__add__`
-- `__radd__`
-- `__sub__`
-- `__rsub__`
-- `__mul__`
-- `__pow__`
-- `__rmul__`
-- `__truediv__`
-- `__rtruediv__`
-- `__lt__`
-- `__rlt__`
-- `__gt__`
-- `__rgt__`
-- `__le__`
-- `__rle__`
-- `__ge__`
-- `__rge__`
-- `__eq__`
-- `__ne__`
-- `__and__`
-- `__rand__`
-- `__or__`
-- `__ror__`
-- `__xor__`
-- `__rxor__`
-
-
-**Unary operators (`DataFrame`, `Series`, and `Index`)**:
-
-- `__invert__`
-- `__neg__`
-- `__pos__`
-
-**Accessors**:
-
-- `CategoricalAccessor`
-- `DatetimeAccessor`
-- `StringAccessor`
-
-**Function**
-
-- `concat`
-- `from_pandas`
-- `merge`
-- `pivot_table`
-- `read_csv`
-- `read_parquet`
-- `repartition`
-- `to_datetime`
-- `to_numeric`
-- `to_timedelta`
-- `to_parquet`
+- ``melt``
+- named GroupBy Aggregations