version 2023.3.0 issue

[yduan-polo]

After upgrade sf3 and fsspectp 2023.3.0 we see the following error, is there a bug or can someone advise how to resolve this? Thanks.

[ERROR] PermissionError: [Errno 1] The specified method is not allowed against this resource.
Traceback (most recent call last):
  File "/var/task/google_analytics/etl.py", line 119, in run_reports
    df.to_csv(get_s3_location(report), index=False)
  File "/var/task/pandas/core/generic.py", line 3551, in to_csv
    return DataFrameRenderer(formatter).to_csv(
  File "/var/task/pandas/io/formats/format.py", line 1180, in to_csv
    csv_formatter.save()
  File "/var/task/pandas/io/formats/csvs.py", line 241, in save
    with get_handle(
  File "/var/task/pandas/io/common.py", line 667, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/var/task/pandas/io/common.py", line 393, in _get_filepath_or_buffer
    file_obj = fsspec.open(
  File "/var/task/fsspec/core.py", line 419, in open
    return open_files(
  File "/var/task/fsspec/core.py", line 283, in open_files
    [fs.makedirs(parent, exist_ok=True) for parent in parents]
  File "/var/task/fsspec/core.py", line 283, in
    [fs.makedirs(parent, exist_ok=True) for parent in parents]
  File "/var/task/fsspec/asyn.py", line 115, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/var/task/fsspec/asyn.py", line 100, in sync
    raise return_result
  File "/var/task/fsspec/asyn.py", line 55, in _runner
    result[0] = await coro
  File "/var/task/s3fs/core.py", line 892, in _makedirs
    await self._mkdir(path, create_parents=True)
  File "/var/task/s3fs/core.py", line 877, in _mkdir
    await self._call_s3("create_bucket", **params)
  File "/var/task/s3fs/core.py", line 347, in _call_s3
    return await _error_wrapper(
  File "/var/task/s3fs/core.py", line 139, in _error_wrapper
    raise err

[martindurant]

It is hard to say - what were you doing?

Please also turn on s3fs logging to get more information.

Note that v2023.4.0 of fsspec and s3fs are now out.

[yduan-polo]

very simple stuff: pandas dataframe.to_csv save to s3 bucket. Before upgrade, we use

S3fs: 0.4.2
Fsspec: 0.8.1

But aws lambda has upgraded runtime boto3 to higher version

boto3 version: 1.24.59
botocore version: 1.23.32

causing import error for s3fs, so we upgraded to 2023.3.0, but getting permision error.

Can you advise how to "turn on s3fs logging "?

[yduan-polo]

One thing that I don't understand and probably is what it caused the error is why s3fs tries to create bucket?

File "/var/task/s3fs/core.py", line 877, in _mkdir
await self._call_s3("create_bucket", **params)

[martindurant]

you could do

import fsspec
fsspec.utils.setup_logging(logger_name="s3fs")

Does the bucket already exist?

[yduan-polo]

Thanks.

Yes, it was working before upgraded to 2023.3.0.

[martindurant]

Does the bucket already exist?

[yduan-polo]

Yes the bucket exists along with the proper permissions that the lambda role has.
Wonder why s3fs wants to create the bucket. Is creating bucket part of the process if it thinks the bucket doesn't exist.