Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TimeOutError when updating Dataset with large files #31

Open
kbrueckmann opened this issue Oct 2, 2024 · 0 comments
Open

TimeOutError when updating Dataset with large files #31

kbrueckmann opened this issue Oct 2, 2024 · 0 comments

Comments

@kbrueckmann
Copy link

Dataverse version: 6.3 build v6.3+10797_dls
Python: 3.12

When updating a dataset with the add_file() and update()-methods the uploader falls back to the native API and then works perfectly for smaller files (ca. 50 MB). But if I try to do the same for large files (ca. 20 GB), I get a TimeOutError.

Using the API and curl directly works fine, though; so it doesn't look like it's connected to a timeout of our payara/server. Did anyone have the same problem and has any ideas for a solution?

Here is the complete error message:

Traceback (most recent call last):
  File "...updateDataset.py", line 108, in <module>
    update_dataset(dataset, ".", args.desc if args.desc else "", categories if args.categories else ["Data"])
  File "...updateDataset.py", line 68, in update_dataset
    dataset.update()
  File "...venv/lib/python3.12/site-packages/easyDataverse/dataset.py", line 237, in update
    update_dataset(
  File "...venv/lib/python3.12/site-packages/easyDataverse/uploader.py", line 128, in update_dataset
    _uploadFiles(
  File "...venv/lib/python3.12/site-packages/easyDataverse/uploader.py", line 91, in _uploadFiles
    dvuploader.upload(
  File "...venv/lib/python3.12/site-packages/dvuploader/dvuploader.py", line 120, in upload
    asyncio.run(
  File "...venv/lib/python3.12/site-packages/nest_asyncio.py", line 30, in run
    return loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...venv/lib/python3.12/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
           ^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/usr/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "...venv/lib/python3.12/site-packages/dvuploader/nativeupload.py", line 78, in native_upload
    responses = await asyncio.gather(*tasks)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
    future.result()
  File "/usr/lib/python3.12/asyncio/tasks.py", line 316, in __step_run_and_handle_result
    result = coro.throw(exc)
             ^^^^^^^^^^^^^^^
  File "...venv/lib/python3.12/site-packages/dvuploader/nativeupload.py", line 211, in _single_native_upload
    async with session.post(endpoint, data=formdata) as response:
  File "...venv/lib/python3.12/site-packages/aiohttp/client.py", line 1355, in __aenter__
    self._resp: _RetType = await self._coro
                           ^^^^^^^^^^^^^^^^
  File "...venv/lib/python3.12/site-packages/aiohttp/client.py", line 686, in _request
    await resp.start(conn)
  File "...venv/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1014, in start
    with self._timer:
  File "...venv/lib/python3.12/site-packages/aiohttp/helpers.py", line 719, in __exit__
    raise asyncio.TimeoutError from None
TimeoutError

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant