-
-
Notifications
You must be signed in to change notification settings - Fork 18.6k
EOFError on Gzipped CSV read from S3 #28206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Are you getting the same issue if you save the file locally and don't use s3? |
I get no error when reading the same file locally. I just tested with a few different versions of s3fs and it works in 0.2.2, but then fails with the EOFError in version 0.3.0 and up. |
Can you decompress the file using s3fs alone (no pandas)? |
Good question! No, s3fs+gzip get the same error: import gzip
import s3fs
fs = s3fs.S3FileSystem()
s3_fh = fs.open('s3://bucketname/file.csv.gz')
fh = gzip.open(s3_fh)
data = fh.read()
So looks like an s3fs issue. I'll close this one. Thanks for the help @WillAyd ! |
Not sure if this is a Pandas issue or s3fs issue:
Gives the following error:
Setup
Installed Pandas and s3fs via pip:
Pandas version: 0.25.1
s3fs version: 0.3.3
The text was updated successfully, but these errors were encountered: