s3.to_csv ignores pandas_kwargs when dataset is set to True

[jasadams]

if dataset = True as an argument to s3.to_csv, than any additional arguments passed are ignored. e.g. compression='gzip' which I wanted to be passed to pandas.to_csv does not happen

[igorborgest]

Hi @jasadams,

Unfortunately Pandas does not support in-memory compression for CSV files. So we will not support it by now because we need to serialize it in memory during the S3 upload.

Original Issue: pandas-dev/pandas#22555

Adding a couple of notes in the docs to address this issue by now.

Note
----
If `dataset=True`, `pandas_kwargs` will be ignored due
restrictive quoting, date_format, escapechar, encoding, etc required by Athena/Glue Catalog.

Note
----
By now Pandas does not support in-memory CSV compression.
https://github.com/pandas-dev/pandas/issues/22555
So the `compression` will not be supported on Wrangler too.

[igorborgest]

Hi @jasadams

Pandas 1.2.0 is available so we added support to it in the PR above 👆 .
Could you give it a try before the official release? You can install that directly from the dev branch:

pip install git+https://github.com/awslabs/aws-data-wrangler.git@write-compressed-text

[igorborgest]

Released on version 2.3.0 🚀