s3fs fetching data from s3 fails while boto3 get_object for the same object works

[l1x]

Env:

❯ ipython
Python 3.11.1 (main, Dec 23 2022, 09:28:24) [Clang 14.0.0 (clang-1400.0.29.202)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.8.0 -- An enhanced Interactive Python. Type '?' for help.

Version:

In [1]: import s3fs

In [2]: s3fs.__version__
Out[2]: '2022.11.0'

In [3]: fsspec.__version__
Out[13]: '2022.11.0'

In [4]: import boto3

In [5]: boto3.setup_default_session(profile_name='datadeft-dev-eu')

In [7]: import fsspec

In [8]: with fsspec.open("s3://hellohexa-ci/iris.csv") as f:
   ...:     print(f)
   ...:

.....

FileNotFoundError: hellohexa-ci/iris.csv

In [10]: s3 = boto3.client("s3")

In [11]: s3.get_object(Bucket="hellohexa-ci", Key="iris.csv").get("ResponseMetadata")
Out[11]:
{'RequestId': 'YJ04NEX2C5E9HPFV',
 'HostId': '8OxPPB4ZFufC+EpSDxnM5o/DJ/SgOHb+bxMXk8Inb4MaWHzAHYZXgxrVPs9E2p4hHhLub+3eVIM=',
 'HTTPStatusCode': 200,
 'HTTPHeaders': {'x-amz-id-2': '8OxPPB4ZFufC+EpSDxnM5o/DJ/SgOHb+bxMXk8Inb4MaWHzAHYZXgxrVPs9E2p4hHhLub+3eVIM=',
  'x-amz-request-id': 'YJ04NEX2C5E9HPFV',
  'date': 'Sun, 08 Jan 2023 14:29:42 GMT',
  'last-modified': 'Sat, 07 Jan 2023 20:55:47 GMT',
  'etag': '"57fce90c81521889c736445f058c4838"',
  'accept-ranges': 'bytes',
  'content-type': 'text/csv',
  'server': 'AmazonS3',
  'content-length': '3716'},
 'RetryAttempts': 0}

Not sure why this is happening.

[martindurant]

You provided a profile to boto but not to s3fs. Try with

with fsspec.open("s3://hellohexa-ci/iris.csv", profile="datadeft-dev-eu") as f:
    print(f)

[l1x]

with fsspec.open("s3://hellohexa-ci/iris.csv", profile="datadeft-dev-eu") as f:
    print(f)

Thanks @martindurant , I got the same results:

FileNotFoundError: hellohexa-ci/iris.csv

[martindurant]

You might want to try

fs = fsspec.filesystem("s3", profile="datadeft-dev-eu")
fs.ls("")

to get an idea of whether your creds are being correctly selected and authenticated. It's also worth having a look at your AWS_* environment variables, .aws and .boto* locations to see if you have anything that might be conflicting.

[martindurant]

#701 (comment) suggests that setting cache_regions=True for S3FileSystem or specifying the region of the bucket as your default region might be what you need. Can you try?