Dev 20240517 to main (#89)

* catch gdrive errors, remove backup on S3 INT

-fix for #86
-removed the backup on S3 INT option with rclone

* Functional util_addproperties.py

Script to add properties to existing GEE assets.
#79

* added MSG LST step0

feat: Add processing pipeline for daily LandSurface mosaics over Switzerland with step0_processor_msg_lst.py

- Introduced a new script to access and upload LandSurface (LST) data over Switzerland to Google Earth Engine (GEE).
- Utilized CMS SAF data provided by MeteoSwiss via s3 on data.geo.admin.ch to store as SATROMO assets and to calculate VCI, TCI, and VHI.
- Added functionality to retrieve and process NetCDF files, extracting relevant attributes and dimensions.
- Implemented the export of NetCDF bands to GeoTIFF format and scaling operations using Rasterio.
- Added functions to generate LST mosaics for specific dates and handle the upload process to Google Cloud Storage and GEE.
- Ensured proper error handling and cleanup of temporary files and datasets.
- added in dev_config a new PRODUCT_MSG which configs the LST
- moved util_addproperties to main_functions

* updated requirements.txt for MSG LST

-Update dependencies in requirements.txt for LST METEOSWISS processor
- removed unnecessary mount point to google cloud

* added MSG LST CLIMA

MSG LST  and MSG LST CLIMA can be calculated with the step0 function as the otehr products. They won't have a step1. This eases reprocessing
-satromo processor: added PRODUCT_MSG_CLIMA and PRODUCT_MSG exit points
-dev_config: added RODUCT_MSG_CLIMA and PRODUCT_MSG

 new
-step0_processor_msg_lst.py
-step0_processor_msg_lst_clima.py

* added command line option for date

we can now reprocess a specific date from command line 'python satromo_processor.py prod_config.py 2024-06-12', the config file is a must when us with date option

* first steps to get rid of the black part in the visualisation

#56
- outside of the swiss perimeter is nan, inside, but in the swiss perimeter but outside the flightpath is zero.
- line 605 in step0_processor_s2_sr.py is causing issues.

* Added satromo_processor.drawio.svg

* Update satromo_processor.drawio

* Update README.md

Added Description for Single Scene Processing via Command Line

* Removes black border

Fix for  #56

* Moved ch_buffer_5000m to assets

* moved ch_buffer_5000m.shp to assets path

* Thumbnail

Thumbnail:
        Fixed aspect ratio for VHI and S2 SR.
        Updated background to accommodate new S2 SR tiles cut to the effective path.
        Changed no data values accordingly.

util_listassets.py:
        Added functionality to list all assets of a collection.

util_moveassets_terrainshadow.py:
        Script to initialize Google Earth Engine (GEE) and Google Drive authentication.
        Converts Day-Of-Year (DOY) to datetime strings.
        Manages the copying and updating of assets within Google Earth Engine.

satromo_publish.py:
        Added detection for VHI cases to use the Swiss buffer.

step0_processor_s2_sr.py:
        Replaced swisswalti with swisssurface.
        Set terrainShadowDetectionPrecalculated to True to use pre-calculated shadows.

step1_processor_vhi.py:
        Updated to use T235959 time format instead of 240000 as per definitions with BAFU and MeteoSwiss.

* Update requirements.txt

set certifi to latest

* Update step0_processor_s2_sr.py

fix missing slash at the end of the shadow terrain collection

* Prepared for deploy to prod

---------

Co-authored-by: Joan Sturm <[email protected]>

README.md

@@ -151,6 +151,70 @@ A new function specifically designed for each personal collections is stored in
 (If you don't need a new step0 processor function, you certainly don't need a new collection...)
 Create the new function in a new file located in step0_processors folder.
 
+## Single Scene Processing via Command Line
+
+### Use Case:
+For cases where certain data was not computed on production for some reason (e.g., missing cloud mask etc.). These instructions are suitable for processing scenes via command line on the local machine using an INT or PROD account from GEE with a local version of the code and writing to PROD of STAC and GEE ASSETS.
+
+### Prerequisites
+
+- **Installation**
+  Install according to the instructions in the GitHub repository.
+
+- **Copy MAIN**
+  It is recommended to copy/check out MAIN with Github to ensure consistency in data processing.
+
+- **Objective**
+  The goal is to process a day, e.g., 2024-06-12, for a product.
+
+- **Configure prod_config.py**
+  Configure prod_config.py so that in case of Windows:
+    - Use GEE INT geetest-credentials-int.secret
+    - Set GDRIVE_SOURCE_DEV = "geedriveINT:"
+    - Set GDRIVE_MOUNT_DEV = r'G:\\'
+    - Ensure corresponding drives are mounted
+
+- **Delete Assets**
+  Delete affected (damaged) assets in GEE step0_collection before recalculating step0.
+
+- **Remove empty_step0.csv asset entry for date**
+  Delete all 'no candidate' entries for products in tools\empty_step0.csv if they are to be recalculated.
+
+- **Set Date**
+  Set "LastSceneDate" of the respective product before the date you want to process, e.g., before 2024-06-12, in tools/last_updates.txt.
+
+- **Start Venv Environment**
+  Start and activate the virtual environment:
+    - Open Command Prompt
+    - Change directory where satromo folder is located 
+      ```
+      c:
+      cd C:\temp\topo-satromo (as an example)
+      .venv\Scripts\activate
+      ```
+
+### Execution
+
+1. For June 12, 2024, process the products as defined in prod_config.py:
+```
+(.venv) C:\temp\topo-satromo>python satromo_processor.py prod_config.py 2024-06-12
+```
+
+2. GEE Asset creation complete and Start Export
+Check after about 60 minutes, if the corresponding assets (x2) are present in GEE.
+```
+(.venv) C:\temp\topo-satromo>python satromo_processor.py prod_config.py 2024-06-12
+```
+
+3. Start Publish Export
+Check after about 60 minutes, if the corresponding files (6x4) are present in Google Drive, or keep running the job below.
+```
+(.venv) C:\temp\topo-satromo>python satromo_publish.py prod_config.py
+```
+
+### Update step0_empty_assets.csv List on GitHub
+
+Update the step0_empty_assets.csv directly on GitHub PROD: https://github.com/swisstopo/topo-satromo/blob/main/tools/step0_empty_assets.csv
 
 ## Technologies
 

assets/ch_buffer_5000m_2056_108.cpg

@@ -0,0 +1 @@
+UTF-8

assets/ch_buffer_5000m_2056_108.prj

@@ -0,0 +1 @@
+PROJCS["CH1903+_LV95",GEOGCS["GCS_CH1903+",DATUM["D_CH1903+",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["False_Easting",2600000.0],PARAMETER["False_Northing",1200000.0],PARAMETER["Scale_Factor",1.0],PARAMETER["Azimuth",90.0],PARAMETER["Longitude_Of_Center",7.43958333333333],PARAMETER["Latitude_Of_Center",46.9524055555556],UNIT["Meter",1.0]]

assets/ch_buffer_5000m_2056_108.qmd

@@ -0,0 +1,27 @@
+<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
+<qgis version="3.36.3-Maidenhead">
+  <identifier></identifier>
+  <parentidentifier></parentidentifier>
+  <language></language>
+  <type></type>
+  <title></title>
+  <abstract></abstract>
+  <links/>
+  <dates/>
+  <fees></fees>
+  <encoding></encoding>
+  <crs>
+    <spatialrefsys nativeFormat="Wkt">
+      <wkt></wkt>
+      <proj4></proj4>
+      <srsid>0</srsid>
+      <srid>0</srid>
+      <authid></authid>
+      <description></description>
+      <projectionacronym></projectionacronym>
+      <ellipsoidacronym></ellipsoidacronym>
+      <geographicflag>false</geographicflag>
+    </spatialrefsys>
+  </crs>
+  <extent/>
+</qgis>

assets/ch_buffer_5000m_2056_22.cpg

@@ -0,0 +1 @@
+UTF-8

assets/ch_buffer_5000m_2056_22.prj

@@ -0,0 +1 @@
+PROJCS["CH1903+_LV95",GEOGCS["GCS_CH1903+",DATUM["D_CH1903+",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["False_Easting",2600000.0],PARAMETER["False_Northing",1200000.0],PARAMETER["Scale_Factor",1.0],PARAMETER["Azimuth",90.0],PARAMETER["Longitude_Of_Center",7.43958333333333],PARAMETER["Latitude_Of_Center",46.9524055555556],UNIT["Meter",1.0]]

assets/ch_buffer_5000m_2056_22.qmd

@@ -0,0 +1,27 @@
+<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
+<qgis version="3.36.3-Maidenhead">
+  <identifier></identifier>
+  <parentidentifier></parentidentifier>
+  <language></language>
+  <type></type>
+  <title></title>
+  <abstract></abstract>
+  <links/>
+  <dates/>
+  <fees></fees>
+  <encoding></encoding>
+  <crs>
+    <spatialrefsys nativeFormat="Wkt">
+      <wkt></wkt>
+      <proj4></proj4>
+      <srsid>0</srsid>
+      <srid>0</srid>
+      <authid></authid>
+      <description></description>
+      <projectionacronym></projectionacronym>
+      <ellipsoidacronym></ellipsoidacronym>
+      <geographicflag>false</geographicflag>
+    </spatialrefsys>
+  </crs>
+  <extent/>
+</qgis>

assets/ch_buffer_5000m_2056_65.cpg

@@ -0,0 +1 @@
+UTF-8

assets/ch_buffer_5000m_2056_65.prj

@@ -0,0 +1 @@
+PROJCS["CH1903+_LV95",GEOGCS["GCS_CH1903+",DATUM["D_CH1903+",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["False_Easting",2600000.0],PARAMETER["False_Northing",1200000.0],PARAMETER["Scale_Factor",1.0],PARAMETER["Azimuth",90.0],PARAMETER["Longitude_Of_Center",7.43958333333333],PARAMETER["Latitude_Of_Center",46.9524055555556],UNIT["Meter",1.0]]

assets/ch_buffer_5000m_2056_65.qmd

@@ -0,0 +1,27 @@
+<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
+<qgis version="3.36.3-Maidenhead">
+  <identifier></identifier>
+  <parentidentifier></parentidentifier>
+  <language></language>
+  <type></type>
+  <title></title>
+  <abstract></abstract>
+  <links/>
+  <dates/>
+  <fees></fees>
+  <encoding></encoding>
+  <crs>
+    <spatialrefsys nativeFormat="Wkt">
+      <wkt></wkt>
+      <proj4></proj4>
+      <srsid>0</srsid>
+      <srid>0</srid>
+      <authid></authid>
+      <description></description>
+      <projectionacronym></projectionacronym>
+      <ellipsoidacronym></ellipsoidacronym>
+      <geographicflag>false</geographicflag>
+    </spatialrefsys>
+  </crs>
+  <extent/>
+</qgis>

assets/ch_buffer_5000m_2056_8.cpg

@@ -0,0 +1 @@
+UTF-8

assets/ch_buffer_5000m_2056_8.prj

@@ -0,0 +1 @@
+PROJCS["CH1903+_LV95",GEOGCS["GCS_CH1903+",DATUM["D_CH1903+",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["False_Easting",2600000.0],PARAMETER["False_Northing",1200000.0],PARAMETER["Scale_Factor",1.0],PARAMETER["Azimuth",90.0],PARAMETER["Longitude_Of_Center",7.43958333333333],PARAMETER["Latitude_Of_Center",46.9524055555556],UNIT["Meter",1.0]]

assets/ch_buffer_5000m_2056_8.qmd

@@ -0,0 +1,27 @@
+<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
+<qgis version="3.36.3-Maidenhead">
+  <identifier></identifier>
+  <parentidentifier></parentidentifier>
+  <language></language>
+  <type></type>
+  <title></title>
+  <abstract></abstract>
+  <links/>
+  <dates/>
+  <fees></fees>
+  <encoding></encoding>
+  <crs>
+    <spatialrefsys nativeFormat="Wkt">
+      <wkt></wkt>
+      <proj4></proj4>
+      <srsid>0</srsid>
+      <srid>0</srid>
+      <authid></authid>
+      <description></description>
+      <projectionacronym></projectionacronym>
+      <ellipsoidacronym></ellipsoidacronym>
+      <geographicflag>false</geographicflag>
+    </spatialrefsys>
+  </crs>
+  <extent/>
+</qgis>

configuration/__init__.py

@@ -1,38 +1,48 @@
 import os
 import sys
 import importlib
-# we use importlib from standard library to customize how we load the configuration
+from datetime import datetime
 
-"""
+# Global variable to hold the argument date string
+arg_date_str = None
 
-The first command-line argument is used to define which configuration is loaded.
-The value has to correspond to a python filename found in the configuration directory.
-For example, to launch the program satromo_processor with the specific_config.py file, use:
-
-> python satromo_processor.py specific_config.py
-
-
-By default, the file dev_config.py will be loading as configuration.
-"""
+# The first command-line argument is used to define which configuration is loaded.
+# The value has to correspond to a python filename found in the configuration directory.
+# For example, to launch the program satromo_processor with the specific_config.py file, use:
+# > python satromo_processor.py specific_config.py 2024-06-12
 
+# By default, the file dev_config.py will be loaded as configuration.
 
 if len(sys.argv) > 1:
-    # If a configuration provided in command line argument, try to load the configration
-    print('configuration file specified as command line argument')
+    # If a configuration is provided in command line arguments, try to load the configuration
+    print('Configuration file specified as command line argument')
     config_filename = sys.argv[1]
 
-    # current direct, i.e. configuration
+    # Check if a date is provided
+    if len(sys.argv) > 2:
+        date_arg = sys.argv[2]
+        try:
+            # Validate and parse the date argument
+            date = datetime.strptime(date_arg, "%Y-%m-%d").date()
+            arg_date_str = date.strftime("%Y-%m-%d")
+            print(f'Date argument is valid: {arg_date_str}')
+        except ValueError:
+            print('Invalid date format. Please use YYYY-MM-DD.')
+            sys.exit(1)
+    else:
+        print('No date argument provided.')
+
+    # current directory, i.e., configuration
     directory = os.path.dirname(os.path.realpath(__file__))
-    # construction of the absolute filepath of the targeted configuration file.
+    # construct the absolute filepath of the targeted configuration file.
     config_filepath = os.path.join(directory, config_filename)
 
     if os.path.exists(config_filepath):
         # The specified file is loaded as module
         config_filemodule = os.path.splitext(config_filename)[0]
         module = importlib.import_module('configuration.{}'.format(config_filemodule), package='configuration')
 
-
-        # We simulate here a file load similar to 'from configuration import *'.
+        # Simulate here a file load similar to 'from configuration import *'.
         # This way, the original code using this statement does not need to be modified.
         def __getattr__(name):
             return getattr(module, name)
@@ -51,4 +61,5 @@ def __dir__():
     print('Loading dev_config as default configuration.')
     from .dev_config import *
 
-
+# Export the arg_date_str variable for use in other modules
+__all__ = ['arg_date_str']

configuration/dev_config.py

@@ -25,6 +25,7 @@
 GDRIVE_MOUNT_DEV = r'Y:\\'
 # under Windows, add \\ to escape the backslash like r'X:\\'
 S3_DESTINATION_DEV = r'X:\\'
+
 # GITHUB
 GDRIVE_SOURCE_INT = "geedriveINT:"
 GDRIVE_MOUNT_INT = "localgdrive"
@@ -47,7 +48,7 @@
 
 # Desired buffer in m width around ROI, e.g., 25000, this defines the final extent
 # TODO: check if needed in context with step0
-BUFFER = os.path.join("tools", "ch_buffer_5000m.shp")
+BUFFER = os.path.join("assets", "ch_buffer_5000m.shp")
 OVERVIEW_LAKES = os.path.join("assets", "overview_lakes_2056.shp")
 OVERVIEW_RIVERS = os.path.join("assets", "overview_rivers_2056.shp")
 WARNREGIONS = os.path.join("assets", "warnregionen_vhi_2056.shp")
@@ -100,8 +101,30 @@
     "missing_data": 110,
     'NDVI_reference_data': 'projects/satromo-prod/assets/col/1991-2020_NDVI_SWISS',
     'LST_reference_data': 'projects/satromo-prod/assets/col/2012-2020_LST_SWISS',
-    #"step1_collection": 'projects/satromo-int/assets/VHI_SWISS',
-    #"step0_collection": "projects/satromo-int/assets/COL_S2_SR_HARMONIZED_SWISS"
+    "step1_collection": 'projects/satromo-int/assets/VHI_SWISS',
+    "step0_collection": "projects/satromo-int/assets/COL_S2_SR_HARMONIZED_SWISS"
+}
+
+# MSG – MeteoSchweiz
+PRODUCT_MSG = {
+    #
+    # this is  placeholder, needed for the step0 function,
+    "image_collection": "METEOSCHWEIZ/MSG",
+    "temporal_coverage": 7,  # Days
+    "product_name": "ch.meteoschweiz.landoberflaechentemperatur",
+    "no_data": 0,
+    # 'step0_collection': 'projects/satromo-int/assets/LST_SWISS'
+}
+
+# MSG – MeteoSchweiz: only used for repreocessing
+PRODUCT_MSG_CLIMA = {
+    #
+    # this is  placeholder, needed for the step0 function,
+    "image_collection": "METEOSCHWEIZ/MSG",
+    "temporal_coverage": 7,  # Days
+    "product_name": "ch.meteoschweiz.landoberflaechentemperatur",
+    "no_data": 0,
+    # 'step0_collection': 'projects/satromo-int/assets/LST_CLIMA_SWISS'
 }
 
 # TEST datasets
@@ -252,8 +275,15 @@
     'projects/satromo-int/assets/COL_S3_TOA_SWISS': {
         'step0_function': 'step0_processor_s3_toa.generate_s3_toa_mosaic_for_single_date'
         # cleaning_older_than: 2 # entry used to clean assets
+    },
+    'projects/satromo-int/assets/LST_SWISS': {
+        'step0_function': 'step0_processor_msg_lst.generate_msg_lst_mosaic_for_single_date'
+        # cleaning_older_than: 2 # entry used to clean assets
+    },
+    'projects/satromo-int/assets/LST_CLIMA_SWISS': {
+        'step0_function': 'step0_processor_msg_lst_clima.generate_msg_lst_mosaic_for_single_date'
+        # cleaning_older_than: 2 # entry used to clean assets
     }
-
 }