proposal for a cache file generator

[jgriesfeller]

Using pre-calculated cache files for aeroval analyses can bring down running times quite a bit.

As a part of my testing I more or less have developed a very basic version of cache file generator.

Generating the cache files can be easily parallelized via the queues. We could also put cache files at a central location where they can be automatically regenerated (e.g. weekly) and copied by the users before running an aeroval analysis.

Is there any interest in using something like that?

[avaldebe]

I would like to see your cache implementation.
Could you show it on the next dev meeting?

[jgriesfeller]

It's very simple: just read the data via the API. Example for some EBASMC variables:

from pyaerocom import const
from pyaerocom.io import ReadUngridded

def main():
    DATA_ID = const.EBAS_MULTICOLUMN_NAME
    reader = ReadUngridded(DATA_ID)
    vars_to_cache = [
        "concso4pm25",
        "concso4pm10",
        "concpm10",
        "concpm25",
    ]

    for var_to_read in vars_to_cache:
        data = reader.read(vars_to_retrieve=var_to_read)
        print(f"# of unique stations: {len(data.unique_station_names)}")
        print(data)

if __name__ == "__main__":
    main()

This writes the cache file for every variable noted in vars_to_cache. Using the PROVIDES_VARIABLES constant in every reader class, this can be done for all variables.
The code also just reads the cache files if regenerating the cache is not necessary.