AccessDenied (403) when downloading datasets from psb2-datasets.s3.amazonaws.com causes invalid JSON files

[P4o1o]

Hello,
I'm using psb2.fetch_examples(...) to automatically download datasets from the PSB2 suite. However, the download fails silently due to an access error from the S3 bucket:

• The code uses requests.get(s3_url) without checking the status_code. Therefore, even when AWS responds with HTTP 403 Forbidden, the error XML (AccessDenied) is saved to a file.

• Later, the function load_json_lines(...) attempts to parse this file, which contains XML instead of JSON, leading to a JSONDecodeError.

I've verified with curl -v: the response is indeed a 403 with the AccessDenied error from S3.

Steps to Reproduce

import psb2
train, test = psb2.fetch_examples("datasets/psb2", "bouncing-balls", 2000, 200)
# -> JSONDecodeError, because the file contains AccessDenied XML, not JSON.

[thelmuth]

I'm sorry for the delay in getting to this, it slipped off of my radar.

I'm not having trouble with accessing the files using curl; the following command worked just fine to download the file:

curl -O https://psb2-datasets.s3.amazonaws.com/PSB2/datasets/basement/basement-edge.edn

I have not tried in Python yet, and will soon.

[thelmuth]

I just tested installing psb2 with PIP and running the code above, and it all worked. Can you try again, and let me know if still having the issue?

[P4o1o]

Hi, sorry for the delay and thanks for checking.

curl gives me this output:

curl -O https://psb2-datasets.s3.amazonaws.com/PSB2/datasets/basement/basement-edge.edn
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1823  100  1823    0     0   1435      0  0:00:01  0:00:01 --:--:--  1435

but in a fresh Python 3.12.3 virtual environment (library installed via pip3):

import psb2
train, test = psb2.fetch_examples("datasets/psb2", "bouncing-balls", 2000, 20)
# i tried this, but also with different paramethers, i. e.
train, test = psb2.fetch_examples("datasets/psb2", "bouncing-balls", 1, 1)

In every case the process hangs indefinitely.