-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refactor: use threading for concurrent requests (#133)
Not using aiohttp anymore nor nest-asyncio Also changed from multiprocessing to multi threading for get command
- Loading branch information
1 parent
41d07ad
commit c3bd8b2
Showing
47 changed files
with
1,493 additions
and
1,031 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -76,34 +76,6 @@ python -m pip install copernicusmarine --upgrade | |
|
||
For more comprehensive details on how to use the `copernicusmarine` Toolbox, please refer to our [Help Center](https://help.marine.copernicus.eu/en/collections/9080063-copernicus-marine-toolbox). It ensures a smooth migration for existing users of legacy services such as MOTU, OPeNDAP or FTP. | ||
|
||
### Network configuration | ||
|
||
#### Disable SSL | ||
|
||
A global SSL context is used when making HTTP calls using the `copernicusmarine` Toolbox. For some reason, it can lead to unexpected behavior depending on your network configuration. You can set the `COPERNICUSMARINE_DISABLE_SSL_CONTEXT` environment variable to any value to globally disable the usage of SSL in the toolbox: | ||
|
||
- on **UNIX** platforms: `export COPERNICUSMARINE_DISABLE_SSL_CONTEXT=True` | ||
- on **Windows** platforms: `set COPERNICUSMARINE_DISABLE_SSL_CONTEXT=True` | ||
|
||
#### Trust Env for python libraries | ||
|
||
To do HTTP calls, the Copernicus Marine Toolbox uses two python libraries: requests and aiohttp. By default, those libraries will have `trust_env` values set to `True`. If you want to deactivate this, you can set `COPERNICUSMARINE_TRUST_ENV=False` (default `True`). This can be useful for example if you don't want those libraries to read your `.netrc` file as it has been reported that having a `.netrc` with a line: "default login anonymous password user@site" is incompatible with S3 connection required by the toolbox. | ||
|
||
#### Proxy | ||
|
||
To use proxies, as describe in the [aiohttp documentation](https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support) you can use two options: | ||
|
||
- set the `HTTPS_PROXY` variable. For eg: `HTTPS_PROXY="http://user:[email protected]"`. It should work even with `COPERNICUSMARINE_TRUST_ENV=False`. | ||
- use a `.netrc` file but be aware that having a line: "default login anonymous password user@site" is incompatible with S3 connection required by the toolbox. Also note that if you have `COPERNICUSMARINE_TRUST_ENV=True` (the default value) then if `NETRC` environment variable is set with a specified location, the `.netrc` file will be read from the specified location there rather than from `~/.netrc`. | ||
|
||
#### Number of concurrent requests | ||
|
||
The toolbox makes many requests to STAC to be able to parse the full marine data store STAC catalog. For that, it uses asynchronous calls. It can be problematic to do too many requests at the same time. To limit the number of requests at the same time you can use: `COPERNICUSMARINE_MAX_CONCURRENT_REQUESTS`. The default value is `15` and minimum value is `1`. | ||
|
||
Note, that this concerns only the catalog parsing step so the describe command and the start of the get and subset command. It does not apply when downloading files or listing files from the get command or when requesting the data chunks for the subset command. | ||
|
||
For the `get` command, you can use the `COPERNICUSMARINE_GET_CONCURRENT_DOWNLOADS` to set the number of threads open to download in parallel. There are no default value. By default the toolbox uses the python `multiprocessing.pool.ThreadPool`. You can set the environment variable to 0 if you don't want to use the `multiprocessing` library at all, the download will be used only through `boto3`. | ||
|
||
## Command Line Interface (CLI) | ||
|
||
### The `--help` option | ||
|
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.