Steps to add support for other cloud providers

[lefreud]

MC Hub currently only supports OpenStack cloud. In order to eventually add support for more clouds, I made a list of steps required to do so.

In the following instructions, the steps match what needs to be done to add support for Azure but it can also apply to GCP and AWS.

1. Setup MC Hub locally using the steps in the README.

2. Download and store the MC release of Azure in MC Hub's Dockerfile, in addition to the OpenStack release.

   mc-hub/Dockerfile

   Lines 47 to 50 in ffc1a17

   	## Download Magic Castle Open Stack release
   	RUN curl -L ${MAGIC_CASTLE_URL} -o magic_castle-openstack.zip && \
   	unzip magic_castle-openstack.zip && \
   	rm magic_castle-openstack.zip

3. Modify the configuration.json file to add the authentication environment variables required by Azure to work. This wasn't needed for OpenStack because authentication is done through a file (clouds.yaml), instead of environment variables. I recommend structuring the additionnal configuration the following way.

   {
       ...
       "cloud_provider": "azure",
       "cloud_providers": {
           "openstack": {},
           "azure": {
               "environment_variables": {
                   "ARM_CLIENT_ID": "",
                   "ARM_CLIENT_SECRET": "",
                   "ARM_SUBSCRIPTION_ID": "",
                   "ARM_TENANT_ID": ""
               }
           }
       }
   }

   Refer to Terraform's documentation for more details on authentication with Azure.

   Notice that I added the cloud_provider key too. This will be used later to let MC Hub know whether we are using OpenStack or Azure when parsing the Terraform state file and fetching cloud resources.

4. Create a class called AzureManager, similar to OpenStackManager. The following public functions in OpenStackManager should be implemented by AzureManager. This will require communication with the Azure API through an SDK or direct API calls.

   __init__(self, *, ...)
   test_connection()
   get_available_resources(self)
   

5. Call test_connection of AzureManager on the start of the application. The following code needs to be updated:

   mc-hub/app/server.py

   Lines 11 to 12 in ffc1a17

   	# Exit with an error if the clouds.yaml is not found or the OpenStack API can't be reached
   	OpenStackManager.test_connection()

   Here, a method in CloudManager called test_connection could be added to test the connection of the OpenStack API or Azure API, depending on which one is selected in the configuration.json file. MC Hub stores the content of configuration.json in the config dictionary. This configuration variable can be accessed this way:

   from models.configuration import config
   print(config["cloud_provider"])

6. In order to fetch the available resources from the right cloud provider, the CloudManager class needs to be modified to retrieve the available resources from AzureManager or OpenStackManager, depending on config["cloud_provider"]. The following code needs to be updated:

   mc-hub/app/models/cloud/cloud_manager.py

   Lines 5 to 14 in ffc1a17

   	class CloudManager:
   	def __init__(self, **kwargs):
   	self.__openstack_manager = OpenStackManager(**kwargs)
   	def get_available_resources(self):
   	"""
   	Retrieves the available cloud resources including resources from OpenStack
   	and availables domains.
   	"""
   	available_resources = self.__openstack_manager.get_available_resources()

7. Because Azure uses environment variables for authentication, we also need a method for getting these environment variables from the configuration.json file. Create a method called get_environment_variables in AzureManager.

8. When running terraform plan with Azure, the environment variables from Azure need to be passed to Terraform by using the method created previously.

   mc-hub/app/models/magic_castle/magic_castle.py

   Lines 290 to 292 in ffc1a17

   	environment_variables = environ.copy()
   	dns_manager = DnsManager(self.get_domain())
   	environment_variables.update(dns_manager.get_environment_variables())

9. The same environment variables need to be passed when running terraform apply.

   mc-hub/app/models/magic_castle/magic_castle.py

   Lines 342 to 344 in ffc1a17

   	environment_variables = environ.copy()
   	dns_manager = DnsManager(self.get_domain())
   	environment_variables.update(

10. The class MagicCastleConfigurationSchema, which essentially validates the schema format of the data that will be used in the main.tf.json file, will need to be modified.

    I recommend creating an abstract class MagicCastleBaseSchema which contains the configuration common to all clouds. This includes the cluster_name entry, domain, instances, etc. Some entries should be extended by cloud-specific schemas. For instance, the MagicCastleOpenStackSchema would contain a specific image and os_floating_ips. And, the MagicCastleAzureSchema would contain the image key and the location key.

11. The TerraformStateParser will also need to be split in two classes: TerraformOpenStackStateParser and TerraformAzureStateParser in order to look for the right entries in the terraform.tfstate file.

12. Then, the MagicCastleConfiguration class will need to be updated to load, validate and dump the configuration using the right schema and the right TerraformStateParser, depending on the selected cloud.

13. Unit and integration tests for the backend will need to be added and modified to work with Azure.

14. The ClusterEditor component in the Vue.Js frontend will need to be modified to support Azure configurations. This includes supporting the location and image key from Azure and hiding the os_floating_ips combobox.