Use application credentials

items/openstack-backups/README.md

@@ -1,6 +1,6 @@
 # Automation of OpenStack backups
 
-The OpenStack Horizon UI allows the manual creation of backups and snapshots from individual instances and volumnes, but does not provide automation features for backing up or restoring multiple resources nor it offers options for the scheduling of backups. The OpenStack Command Line Interface (CLI) provides the same capabilities, without any automation features. The scheduling of backups using the OpenStack CLI must be done using `cron` jobs. Documentation on how to use the OpenStack CLI to create and schedule backups can be found [here](https://confluence.ecmwf.int/display/EWCLOUDKB/EWC+OpenStack+API+access+-+How+to+create+backups+from+VMs) and to restore backups [here](https://confluence.ecmwf.int/display/EWCLOUDKB/EWC+OpenStack+API+access+-+How+to+restore+backups+from+VMs). With this script, users can create, schedule or restore multiple backups automatically.
+The OpenStack Horizon UI allows the manual creation of backups and snapshots from individual instances and volumes, but does not provide automation features for backing up or restoring multiple resources nor it offers options for the scheduling of backups. The OpenStack Command Line Interface (CLI) provides the same capabilities, without any automation features. The scheduling of backups using the OpenStack CLI must be done using `cron` jobs. Documentation on how to use the OpenStack CLI to create and schedule backups can be found [here](https://confluence.ecmwf.int/display/EWCLOUDKB/EWC+OpenStack+API+access+-+How+to+create+backups+from+VMs) and to restore backups [here](https://confluence.ecmwf.int/display/EWCLOUDKB/EWC+OpenStack+API+access+-+How+to+restore+backups+from+VMs). With this script, users can create, schedule or restore multiple backups automatically.
 
 ## Functionality
 This script automates the creation, restoration and scheduling of backups using the OpenStack SDK. This allows users with OpenStack credentials to create backups or snapshots of multiple instances and their attached volumes, scheduling backups for a future time with and without repetition and with and without a retention count. It also allows the restoration of multiple instances or volumes, in-place or to a new instance or volume.
@@ -17,39 +17,18 @@ OR
 * `docker>=29.1.3`
 
 ## Usage
-The script can be run either directly using the current python environment, satisfying the prerequisites above, or inside a docker container. In either case, it is necessary to first set up the authentication credentials file and the configuration file.
-
-### 1. Authentication credentials
-
-It is necessary to have the required OpenStack credentials to access the project/domain/cloud specified in the configuration file. The program expects a credentials file in the root directory called `clouds.yaml`, which contains the necessary information for authentication into the cloud. An example authentication file to acess in to `my_cloud` with `my_username` is
-```
-clouds:
-    my_cloud:
-        auth_type: v3oidcpassword
-        auth:
-            auth_url: https://keystone.cloudferro.com:5000/v3
-            username: my_username
-            password: my_password
-            project_id: my_project_id
-            project_name: my_project_name
-            project_domain_name: my_domain_name
-            project_domain_id: my_project_domain_id
-            client_id: openstack
-            client_secret: my_client_secret
-            protocol: openid
-            identity_provider: eumetsat_provider
-            discovery_endpoint: https://identity.cloudferro.com/auth/realms/Eumetsat-elasticity/.well-known/openid-configuration
-        region_name: WAW3-1
-        interface: public
-        identity_api_version: 3
-```
-The `clouds.yaml` file can be obtained from the cloud server provider or filled manually with the information in an OpenStack RC file. A template `clouds.yaml` file can be found in the `templates` directory.
+The script can be run either directly using the current python environment, satisfying the prerequisites above, or inside a docker container. In either case, it is necessary to first set up the authentication credentials file and the configuration file. It is highly recommended not to run this script within the VM you wish to back up or restore.
+
+### 1. Application credentials
+
+To run this script it is necessary to have the required OpenStack application credentials to access the project/domain/cloud specified in the configuration file. You can find information on how to create application credentials and obtain the RC file or clouds.yaml file in [here](https://confluence.ecmwf.int/display/EWCLOUDKB/EWC+Cloud+Management+UI+-+Identity+-+Create+Application+Credentials)
 
 ### 2. Configuration file
 
 A configuration YAML file that contains the requested information to create, schedule or restore backups is required to run the script. A template YAML file can be found in the `templates` directory and some examples in the `tests` directory. The structure of this configuration file is as follows
 ```
 cloud: <cloud_name>
+authentication: <credentials_method>
 
 backup:
 - name: <resource_name>
@@ -63,7 +42,7 @@ restore:
   mode: <backup_mode>
 - ...
 ```
-where the `<cloud_name>` corresponds to the name of cloud (domain name) to which the various resources belong. The optional `backup` node contains instructions to create and schedule backups, as explained below, and the optional `restore` node contains instructions to restore backups.
+where the `<cloud_name>` corresponds to the name of cloud (domain name) to which the various resources belong. The `authentication` node indicates the mode in which the credentials will be provided, which should be either `openrc` if the application credentials have been sourced from an OpenRC file, or `clouds.yaml` if the application credentials are stored in an eponymous file in the current directory. The optional `backup` node contains instructions to create and schedule backups, as explained below, and the optional `restore` node contains instructions to restore backups.
 
 
 #### 2.1 Creating backups
@@ -83,7 +62,7 @@ backup:
 - ...
 ```
 
-where `<resource_name>` is the name of the instance or volume to back up, `<resource_type>` is either `instance` or `volume` and `<backup_mode>` is either `snapshot` or `backup`. Any number of entries, corresponding to the resources to backup, can be provided to the `backup` node. By default, attachments are not backed up along with the resource. In order to back these up, one must provide the `attachment` field to the resource, as seen above. It is possible to select to backup all attachments of the resource, with `attachments: all`, or a specific set, by providing a list of the resources to backup. It is recommended to stop instances and detaching volumes before backing them up. This is the default behaviour. The options `stop` and `detach` can be supplied to instances and volumes, respectively, to change this default behaviour.
+where `<resource_name>` is the name of the instance or volume to back up, `<resource_type>` is either `instance` or `volume` and `<backup_mode>` is either `snapshot` or `backup`. Any number of entries, corresponding to the resources to backup, can be provided to the `backup` node. By default, attachments are not backed up along with the resource (with the exception of root volume of volume-backed instances). In order to back these up, one must provide the `attachment` field to the resource, as seen above. It is possible to select to backup all attachments of the resource, with `attachments: all`, or a specific set, by providing a list of the resources to backup. It is recommended to stop instances and detaching volumes before backing them up. This is the default behaviour. The options `stop` and `detach` can be supplied to instances and volumes, respectively, to change this default behaviour.
 
 #### 2.2 Scheduling backups
 

items/openstack-backups/openstack_backups.py

@@ -54,13 +54,14 @@ def parse_config_file(config_file_path: str) -> dict:
             else:
                 raise RuntimeError(f'{yaml_error}: Cloud name missing.')
 
-            # Validate the authentication method (default clouds.yaml)
-            # FIXME: Currently the openrc method does not work, so force it to be clouds.yaml
-            #if "auth" in yaml_contents:
-            #    config['auth'] = yaml_contents['auth']
-            #else:
-            #    config['auth'] = 'openrc'
-            config['auth'] = 'clouds.yaml'
+
+            # Authentication method can be with Application Credentials from an OpenRC file or a clouds.yaml file
+            if 'authentication' not in yaml_contents:
+                raise RuntimeError(f'{yaml_error}: Missing application credentials. The entry \'authentication\' is required.')
+            elif yaml_contents['authentication'] not in ['openrc','clouds.yaml']:
+                raise RuntimeError(f'{yaml_error}: Unknown application credentials: The entry \'authentication\' should be either \'openrc\' or \'clouds.yaml\'.')
+            else:
+                config['authentication'] = yaml_contents['authentication']
 
             # Validate the backup entry
             if 'backup' in yaml_contents:
@@ -152,7 +153,6 @@ def parse_config_file(config_file_path: str) -> dict:
                     restore['in_place'] = restore.get('in_place', False)
 
                     # If the restoration is for an instance and not in place, one needs to provide flavor, network and (optionally) security groups
-                    # TODO: Might restore all networks later
                     if restore['type'] == 'instance' and not restore['in_place']:
                         if 'flavor' not in restore.keys():
                             raise RuntimeError(f'{yaml_error}: Restore entry missing flavor, required for non-in-place restorations.')
@@ -170,26 +170,74 @@ def parse_config_file(config_file_path: str) -> dict:
     else:
         raise FileNotFoundError('Backups config file not found.')
 
-def authenticate(auth: str) -> None:
+def authenticate(authentication: str) -> None:
     """
-    Ensure authentication credential are ready.
+    Ensure application credentials are ready.
     Two methods of authentication, with openrc file (pre-source) or clouds.yaml file
     """
 
-    if auth=="openrc":
-        # Check the appropriate environment variables have been set
-        #required_envs = ['OS_IDENTITY_API_VERSION', 'OS_AUTH_URL', 'OS_AUTH_TYPE', 'OS_USERNAME', 'OS_PASSWORD', 'OS_PROJECT_NAME', 'OS_PROJECT_DOMAIN_ID']
-        required_envs = ['OS_AUTH_URL', 'OS_INTERFACE', 'OS_IDENTITY_API_VERSION', 'OS_USERNAME', 'OS_REGION_NAME', 'OS_USER_DOMAIN_NAME', 'OS_PROJECT_DOMAIN_ID', 'OS_AUTH_TYPE', 'OS_APPLICATION_CREDENTIAL_ID', 'OS_APPLICATION_CREDENTIAL_SECRET']
+    if authentication=="openrc":
+
+        # Check if authentication method uses application credentials of password
+
+        if "OS_AUTH_TYPE" in os.environ:
+
+            required_envs = ['OS_AUTH_TYPE', 'OS_AUTH_URL', 'OS_IDENTITY_API_VERSION', 'OS_REGION_NAME', 'OS_INTERFACE']
 
-        for env in required_envs:
-            if env not in os.environ:
-                raise RuntimeError(f'Authentication not possible. Environment variable {env} not set. Source the openrc file in order to set all required environment varibles')
+            # Application credentials
+            if os.getenv("OS_AUTH_TYPE") == "v3applicationcredential":
+                required_envs += ['OS_APPLICATION_CREDENTIAL_ID', 'OS_APPLICATION_CREDENTIAL_SECRET']
+            elif os.getenv("OS_AUTH_TYPE") == "v3oidcpassword":
+                required_envs += ['OS_USERNAME', 'OS_PASSWORD', 'OS_CLIENT_ID', 'OS_CLIENT_SECRET', 'OS_PROTOCOL', 'OS_IDENTITY_PROVIDER', 'OS_DISCOVERY_ENDPOINT', 'OS_USER_DOMAIN_NAME', 'OS_PROJECT_DOMAIN_ID']
+            else:
+                raise RuntimeError(f'Authentication not possible. Unrecognised authentication type {os.getenv("OS_AUTH_TYPE")}')
+
+            for env in required_envs:
+                if env not in os.environ:
+                    raise RuntimeError(f'Authentication not possible. Environment variable {env} not set. Source the openrc file in order to set all required environment varibles')
+
+        else:
+           raise RuntimeError(f'Authentication not possible. Environment variable {env} not set. Source the openrc file in order to set all required environment varibles')
+
+    elif authentication=='clouds.yaml':
 
-    elif auth=='clouds.yaml':
         # Check the the appropriate clouds.yaml file exists in the current directory
+
         if not os.path.exists('clouds.yaml'):
             raise RuntimeError(f'Authentication not possible. Required file `clouds.yaml` not present in the current directory.')
 
+        # Check contents of clouds.yaml
+        with open("clouds.yaml", 'r') as f:
+
+            clouds_yaml = yaml.safe_load(f)
+
+            if 'clouds' not in clouds_yaml:
+                raise RuntimeError(f'Authentication not possible, missing `clouds` entry in `clouds.yaml` file.')
+
+            clouds = clouds_yaml['clouds']
+            if 'openstack' not in clouds:
+                raise RuntimeError(f'Authentication not possible, missing `openstack` entry in `clouds.yaml` file.')
+
+            openstack = clouds['openstack']
+
+            if 'auth_type' not in openstack:
+                raise RuntimeError(f'Authentication not possible, missing `auth_type` in `clouds.yaml` file.')
+
+            if 'auth' not in openstack or  \
+               'auth_url' not in openstack['auth'] or \
+               'application_credential_id' not in openstack['auth'] or \
+               'application_credential_secret' not in openstack['auth'] :
+                    raise RuntimeError(f'Authenciation not possible, missing application credentials in `clouds.yaml` file.')
+
+            if 'regions' not in openstack:
+                raise RuntimeError(f'Authentication not possible, missing `region` in `clouds.yaml` file.')
+
+            if 'interface' not in openstack:
+                raise RuntimeError(f'Authentication not possible, missing `interface` in `clouds.yaml` file.')
+
+            if 'identity_api_version' not in openstack:
+                raise RuntimeError(f'Authentication not possible, missing `identity_api_version` in `clouds.yaml` file.')
+
     else:
         raise RuntimeError(f'Authentication not possible. Unrecognised authentication metod.')
 
@@ -209,7 +257,7 @@ def main():
         config = parse_config_file(args.config_file_path)
 
         # Ensure authentication credentials are ready
-        authenticate(config['auth'])
+        authenticate(config['authentication'])
 
         # Connect to the cloud
         cloud = openstack_connect(config['cloud'])