Skip to content

Commit 7da4a77

Browse files
antoinetavantantoinetavant
committed
Add S3 download function
1 parent a69841f commit 7da4a77

File tree

4 files changed

+192
-0
lines changed

4 files changed

+192
-0
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,3 +165,4 @@ data/bronze/*.xls
165165
doc/_build
166166
.vscode/settings.json
167167
data/silver/*.nc
168+
data/silver/weather_forecasts/*.nc

‎doc/getting_started/data_sources.rst

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
Data Sources
2+
============
3+
4+
This Page lists the different data sources available, and the corresponding way to access them.
5+
6+
Energy Data Sources
7+
-------------------
8+
9+
Predicted consumption computed by RTE
10+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11+
12+
The total consumption of France is computed by RTE.
13+
14+
The corresponding API is named `Consumption <https://data.rte-france.com/catalog/-/api/consumption/Consumption/v1.2>`_
15+
You can use it to access
16+
17+
- "short_term" : the expected consumption for today, tomorrow (D-1) and the day after tomorrow (D-2)
18+
- "weekly_forecasts" : the expected consumption for D-3 to D-9
19+
20+
When requesting dates in the past, the history of the forecasts is available.
21+
22+
To access it, use the class ``energy_forecast.consumption_forecast.PredictionForecastAPI``
23+
24+
Predicted Productions computed by RTE
25+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
26+
27+
The Solar and Eolian productions in France is computed by RTE.
28+
29+
The corresponding API is named `Generation Forecast <https://data.rte-france.com/catalog/-/api/generation/Generation-Forecast/v2.1>`_
30+
You can use it to access
31+
32+
- "SOLAR" : the expected solar production for today (D-0), and tomorrow (D-1)
33+
- "WIND_ONSHORE" : the expected eolian production for today (D-0), and tomorrow (D-1)
34+
35+
Other production means are available, some with a longer forecast horizon (D-2 to D-3)
36+
37+
When requesting dates in the past, the history of the forecasts is available.
38+
However, the API is limited to 21 days of history per call, so a lot of calls are needed to get a long history.
39+
40+
41+
When requesting dates in the past, the history of the forecasts is available.
42+
43+
To access it, use the class ``energy_forecast.production_forecast.ProductionForecastAPI``
44+
45+
Tempo labels computed by RTE
46+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
47+
48+
The Tempo labels are provided by RTE.
49+
The corresponding API is named `Tempo Like Supply Contract <https://data.rte-france.com/catalog/-/api/consumption/Tempo-Like-Supply-Contract/v1.1>`_
50+
51+
You can use it to access the tempo labels for the current day, and the next day, and all the history of the tempo labels.
52+
53+
To access it, use the class ``energy_forecast.tempo_rte.TempoSignalAPI``
54+
55+
About Eco2mix
56+
-------------
57+
58+
Eco2mix is a service provided by RTE, which provides a lot of data about the energy production and consumption in France.
59+
Some are the same as the ones provided by the APIs above, but some are not.
60+
61+
In particular, the historical regional production and consumption are available.
62+
63+
You can access it using the function ``energy_forecast.eco2mix.get_data`` (Refactoring in progress)
64+
65+
Weather Data Sources
66+
--------------------
67+
68+
Weather forecasts
69+
~~~~~~~~~~~~~~~~~
70+
71+
The weather forecasts are provided by Meteo France with different levels of details, corresponding to different model names.
72+
73+
The raw files can be downloaded from `meteo.data.gouv.fr <https://meteo.data.gouv.fr>`_
74+
75+
You can access the Arpege model forecasts using the class ``energy_forecast.meteo.ArpegeSimpleAPI``
76+
This client class download the data as Grib2 files, which requier Xarray and cfgrib to be read.
77+
78+
Historical weather forecasts
79+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
80+
81+
The historical weather forecasts are not publicly available.
82+
Fortunately, this data as beed stored by a french assosiation of amateur meteorologists.
83+
84+
The interesting parameters has been extected from the raw data, and are stored in ``data/silver/weather``.
85+
86+
To ge the data the first time, you need to download it from S3 using ``energy_forecast.meteo.download_historical_forecasts``.
87+
To do so, the S3 credentials are needed, contact the project's owner.
88+
89+
Up to D+3 forecasts are available from the model Arpege, from 2022-01-01 to 2024-04-01.
90+

‎doc/getting_started/index.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,13 @@ Project description
55
-------------------
66
See the :doc:`project description <project_description>` for a detailed description of the project.
77

8+
9+
Data sources
10+
------------
11+
12+
See the :doc:`data sources <data_sources>` for a detailed description of the data sources available
13+
and the corresponding way to access them.
14+
815
Installation
916
------------
1017
You can install the package from the source code by cloning the repository and
@@ -44,3 +51,4 @@ To see move detailed use cases, you can check the tutorials in the :doc:`../user
4451
:hidden:
4552

4653
project_description
54+
data_sources

‎src/energy_forecast/meteo.py

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -377,6 +377,99 @@ def warm_cache(logger, date=None, max_counter=30, sleep_duration=600):
377377
if counter > max_counter:
378378
raise TimeoutError("Max counter reached")
379379

380+
def download_historical_forecasts(s3_key,
381+
s3_secret,
382+
s3_entrypoint,
383+
s3_bucket,
384+
prefix="./",
385+
variables="all",
386+
forecast_type="all",
387+
dryrun=False
388+
):
389+
"""Download the historical forecasts from the S3 bucket.
390+
391+
Parameters
392+
----------
393+
s3_key : str
394+
the key to access the S3 bucket.
395+
s3_secret : str
396+
the secret to access the S3 bucket.
397+
s3_entrypoint : str
398+
the entrypoint of the S3 bucket.
399+
s3_bucket : str
400+
the name of the S3 bucket.
401+
prefix : str
402+
The prefix where the files are downloaded.
403+
Should be similar to ``"./data/silver"``.
404+
variables : str or list[str], optional
405+
the variables to download.
406+
Can be ``"wind_speed_hourly"``, ``"sun_flux_downward_hourly"``, or ``"temperature_hourly"``
407+
or a list of these values.
408+
Default is ``"all"``, which downloads all the variables.
409+
forecast_type : str or list[str], optional
410+
the forecast type to download.
411+
Can be ``"d0"``, ``"d1"``, ``"d2"``, or ``"d3"``,
412+
or a list of these values.
413+
Default is ``"all"``, which downloads all the forecast types.
414+
dryrun : bool, optional
415+
if True, do not download the files.
416+
Default is False.
417+
418+
Returns
419+
-------
420+
list[Path]
421+
the list of the files downloaded.
422+
"""
423+
import boto3
424+
425+
session = boto3.Session(
426+
aws_access_key_id=s3_key,
427+
aws_secret_access_key=s3_secret,
428+
)
429+
s3 = session.resource("s3", endpoint_url=s3_entrypoint)
430+
bucket = s3.Bucket(s3_bucket)
431+
list_files = []
432+
key_prefix = "weather_forecasts"
433+
if variables == "all":
434+
variables = ["wind_speed_hourly",
435+
"sun_flux_downward_hourly",
436+
"temperature_hourly"]
437+
if isinstance(variables, str):
438+
variables = [variables]
439+
for var in variables:
440+
if var not in ["wind_speed_hourly",
441+
"sun_flux_downward_hourly",
442+
"temperature_hourly"]:
443+
raise ValueError(f"Unknown variable {var} : must be in ['wind_speed_hourly', 'sun_flux_downward_hourly', 'temperature_hourly']")
444+
if forecast_type == "all":
445+
forecast_type = ["d0", "d1", "d2", "d3"]
446+
if isinstance(forecast_type, str):
447+
forecast_type = [forecast_type]
448+
for forecast in forecast_type:
449+
if forecast not in ["d0", "d1", "d2", "d3"]:
450+
raise ValueError(f"Unknown forecast type {forecast} : must be in ['d0', 'd1', 'd2', 'd3']")
451+
452+
for var in variables:
453+
for forecast in forecast_type:
454+
key = f"{key_prefix}/{var}_{forecast}.nc"
455+
# test if the key exists
456+
filename = Path(prefix + "/" + key)
457+
if filename.exists():
458+
print(f"{filename} already downloaded, skipping")
459+
continue
460+
filename.parent.mkdir(parents=True, exist_ok=True)
461+
if dryrun:
462+
print(f"DRY RUN : would Download {key} to {filename}")
463+
# test if the key exists without downloading it
464+
try :
465+
s3.Object(s3_bucket, key).load()
466+
except Exception as e:
467+
print(e)
468+
469+
else:
470+
bucket.download_file(key, filename)
471+
list_files.append(filename)
472+
return list_files
380473

381474

382475
if __name__ == "__main__":

0 commit comments

Comments
 (0)