S3 Endpoint

[timhughes]

Is it possible to pass a s3 endpoint. For example I run a s3 compliant server internally and would like to do something like

import pyarrow.parquet as pq
import s3fs

fs = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url':'https://internal.domain:9000/foo'})
pandas_dataframe = pq.ParquetDataset('s3://your-bucket/', filesystem=fs).read_pandas().to_pandas()

[martindurant]

Certainly. You would do

fs = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 
    "https://internal.domain:9000/foo"})

[timhughes]

Yeah i just think i worked that out.

[bhuiyanmobasshir94]

If anyone wants help for local folder mounting to s3 bucket, can go through this stackoverflow.com/a/77369021/7195890

[martindurant]

@bhuiyanmobasshir94 , note that the SO answer is talking about the other s3fs. This s3fs also allows you to mount a bucket/prefix via FUSE, but performance is poor. For testing purposes, both minio and moto in docker allow you to mount the internal data store to a local directory.