Skip to content

Commit

Permalink
refactor: use threading for concurrent requests (#133)
Browse files Browse the repository at this point in the history
Not using aiohttp anymore nor nest-asyncio
Also changed from multiprocessing to multi threading for get command
  • Loading branch information
renaudjester committed Oct 28, 2024
1 parent e0346ad commit 2b3ec66
Show file tree
Hide file tree
Showing 47 changed files with 1,493 additions and 1,031 deletions.
28 changes: 0 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,34 +76,6 @@ python -m pip install copernicusmarine --upgrade

For more comprehensive details on how to use the `copernicusmarine` Toolbox, please refer to our [Help Center](https://help.marine.copernicus.eu/en/collections/9080063-copernicus-marine-toolbox). It ensures a smooth migration for existing users of legacy services such as MOTU, OPeNDAP or FTP.

### Network configuration

#### Disable SSL

A global SSL context is used when making HTTP calls using the `copernicusmarine` Toolbox. For some reason, it can lead to unexpected behavior depending on your network configuration. You can set the `COPERNICUSMARINE_DISABLE_SSL_CONTEXT` environment variable to any value to globally disable the usage of SSL in the toolbox:

- on **UNIX** platforms: `export COPERNICUSMARINE_DISABLE_SSL_CONTEXT=True`
- on **Windows** platforms: `set COPERNICUSMARINE_DISABLE_SSL_CONTEXT=True`

#### Trust Env for python libraries

To do HTTP calls, the Copernicus Marine Toolbox uses two python libraries: requests and aiohttp. By default, those libraries will have `trust_env` values set to `True`. If you want to deactivate this, you can set `COPERNICUSMARINE_TRUST_ENV=False` (default `True`). This can be useful for example if you don't want those libraries to read your `.netrc` file as it has been reported that having a `.netrc` with a line: "default login anonymous password user@site" is incompatible with S3 connection required by the toolbox.

#### Proxy

To use proxies, as describe in the [aiohttp documentation](https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support) you can use two options:

- set the `HTTPS_PROXY` variable. For eg: `HTTPS_PROXY="http://user:[email protected]"`. It should work even with `COPERNICUSMARINE_TRUST_ENV=False`.
- use a `.netrc` file but be aware that having a line: "default login anonymous password user@site" is incompatible with S3 connection required by the toolbox. Also note that if you have `COPERNICUSMARINE_TRUST_ENV=True` (the default value) then if `NETRC` environment variable is set with a specified location, the `.netrc` file will be read from the specified location there rather than from `~/.netrc`.

#### Number of concurrent requests

The toolbox makes many requests to STAC to be able to parse the full marine data store STAC catalog. For that, it uses asynchronous calls. It can be problematic to do too many requests at the same time. To limit the number of requests at the same time you can use: `COPERNICUSMARINE_MAX_CONCURRENT_REQUESTS`. The default value is `15` and minimum value is `1`.

Note, that this concerns only the catalog parsing step so the describe command and the start of the get and subset command. It does not apply when downloading files or listing files from the get command or when requesting the data chunks for the subset command.

For the `get` command, you can use the `COPERNICUSMARINE_GET_CONCURRENT_DOWNLOADS` to set the number of threads open to download in parallel. There are no default value. By default the toolbox uses the python `multiprocessing.pool.ThreadPool`. You can set the environment variable to 0 if you don't want to use the `multiprocessing` library at all, the download will be used only through `boto3`.

## Command Line Interface (CLI)

### The `--help` option
Expand Down
20 changes: 0 additions & 20 deletions copernicusmarine/aioretry/LICENSE

This file was deleted.

5 changes: 0 additions & 5 deletions copernicusmarine/aioretry/__init__.py

This file was deleted.

162 changes: 0 additions & 162 deletions copernicusmarine/aioretry/retry.py

This file was deleted.

Loading

0 comments on commit 2b3ec66

Please sign in to comment.