chore: Refactor & document the project

README.md

@@ -1,11 +1,90 @@
-# Matacall Test Center
+# MetaCall Test Center
 
-This is a test center for Matacall. It contains a set of test cases for Matacall projects and examples. The test cases are written in a specific yaml format, which is described in the following sections.
-The main script used for testing is `testing.py` and it is mainly used in the CI/CD pipeline of this repository. It can also used to test the projects locally.
+## Overview
 
-## Test Suits Format
+MetaCall Test Center is a comprehensive testing framework designed for MetaCall projects and examples. It provides a structured and efficient way to define, run, and manage test cases across different environments. The primary script, `testing.py`, integrates seamlessly into CI/CD pipelines and supports local testing. This project adheres to best practices, SOLID principles, and design patterns to ensure maintainability, scalability, and ease of contribution.
+
+## Project Structure
+
+The project is organized as follows:
+
+``` bash
+.
+├── README.md
+├── LICENSE
+├── requirements.txt
+├── testing
+│   ├── __init__.py
+│   ├── deploy_manager.py
+│   ├── logger.py
+│   ├── repo_manager.py
+│   ├── runner
+│   │   ├── cli_interface.py
+│   │   ├── faas_interface.py
+│   │   ├── interface_factory.py
+│   │   └── runner_interface.py
+│   ├── test_runner.py
+│   └── test_suites_extractor.py
+├── testing.py
+└── test-suites
+    └── test<example name>.yaml  
+```
+
+### Components
+
+- **`testing.py`**: The main script that orchestrates the testing process by interacting with various components.
+
+- **`deploy_manager.py`**: Manages the deployment of MetaCall projects locally or remotely, ensuring the necessary environments are set up for testing.
+
+- **`logger.py`**: Provides a centralized logging mechanism with configurable verbosity levels, helping to debug and monitor the testing process.
+
+- **`repo_manager.py`**: Handles cloning and managing the code repositories required for testing, ensuring that the latest code is always used.
+
+- **`test_runner.py`**: The core component responsible for executing test cases across different environments by leveraging the strategy pattern for flexibility.
+
+- **`test_suites_extractor.py`**: Extracts test cases from YAML files, ensuring that the test cases are correctly parsed and ready for execution.
+
+- **`runner`**: Contains specific implementations for running tests in different environments:
+  - **`runner_interface.py`**: Defines the interface for all runner implementations, adhering to the Dependency Inversion principle.
+  - **`cli_interface.py`**: Implements the interface for running tests in a CLI environment.
+  - **`faas_interface.py`**: Implements the interface for running tests in a Function-as-a-Service (FaaS) environment.
+  - **`interface_factory.py`**: A factory class that creates instances of the appropriate runner interface based on the environment.
+
+## How It Works
+
+1. **Test Suite Definition**: Test cases are defined in YAML format within the `test-suites` directory. Each test suite specifies the project, repository URL, code files, and individual test cases.
+
+2. **Test Execution**: The `testing.py` script is executed with various command-line arguments. The script then:
+   - Parses the command-line arguments.
+   - Extracts test cases from the specified YAML file.
+   - Clones the repository if not already present.
+   - Deploys the project as a local FaaS (if required).
+   - Runs the test cases across the specified environments (CLI, FaaS, etc.).
+
+3. **Output and Logging**: The results of the test cases are logged based on the specified verbosity level, and any errors encountered during the process are reported.
+
+## Design Choices and Principles
+
+This project adheres to several key design principles and patterns:
+
+- **SOLID Principles**:
+  - **Single Responsibility Principle**: Each class has a single responsibility, making the code easier to understand and maintain.
+  - **Open/Closed Principle**: The code is open for extension but closed for modification. New runner environments can be added without modifying existing code.
+  - **Liskov Substitution Principle**: Subtypes (`CLIInterface`, `FaaSInterface`) can be used interchangeably with their base type (`RunnerInterface`) without affecting the correctness of the program.
+  - **Interface Segregation Principle**: The `RunnerInterface` provides a minimal set of methods required by all runner types, preventing unnecessary dependencies.
+  - **Dependency Inversion Principle**: High-level modules (e.g., `TestRunner`) do not depend on low-level modules (`CLIInterface`, `FaaSInterface`), but both depend on abstractions (`RunnerInterface`).
+
+- **Design Patterns**:
+  - **Factory Pattern**: The `InterfaceFactory` class encapsulates the creation of runner interfaces, promoting flexibility and adherence to the Open/Closed Principle.
+  - **Singleton Pattern**: The `DeployManager` and `RepoManager` classes are implemented as singletons to ensure that only one instance exists throughout the application, avoiding redundant deployments or repository clones.
+  - **Strategy Pattern**: The `TestRunner` uses different strategies (`CLIInterface`, `FaaSInterface`) to run tests in various environments, making the code flexible and easy to extend.
+
+## Usage
+
+### Test Suite Format
+
+Test suites are written in YAML format. Below is an example:
 
-The test suits are written in a yaml format. The following is an example of a test suit for the [random-password-generator-example](https://github.com/metacall/random-password-generator-example)
 ```yaml
 project: random-password-generator-example
 repo-url: https://github.com/metacall/random-password-generator-example
@@ -20,23 +99,36 @@ code-files:
         expected-pattern: 'missing 1 required positional argument'
 ```
 
-## Arguments
+### Running Tests
 
-The following arguments are available for the `testing.py` script:
+To run the tests, use the following command:
 
 ```bash
-> python3 ./testing.py -h
-usage: testing.py [-h] [-V] [-f FILE]
-
-options:
-  -h, --help            show this help message and exit
-  -V, --verbose         increase output verbosity
-  -f FILE, --file FILE  the test suite file name
-  -e, --envs            the environments to run the tests on, e,g: -e faas cli, default is cli
+python3 ./testing.py -f <test-suite-file> -V -e <environments>
 ```
 
-## Example
+- `-f`, `--file`: Specifies the test suite file name.
+- `-V`, `--verbose`: Increases output verbosity.
+- `-e`, `--envs`: Specifies the environments to run the tests on (e.g., `cli`, `faas`).
+
+Example:
 
 ```bash
-python3 ./testing.py -f test-suites/test-time-app-web.yaml -V -e cli
-```
+python3 ./testing.py -f test-suites/test-time-app-web.yaml -V -e cli faas
+```
+
+## Contributing
+
+We welcome contributions to the MetaCall Test Center! Here are a few ways you can help improve the project:
+
+- **Enhance Test Coverage**: Add new test cases or improve existing ones to cover more scenarios.
+- **Optimize Code**: Refactor and optimize the codebase to improve performance and readability.
+- **Extend Functionality**: Implement support for additional environments or enhance existing ones.
+- **Documentation**: Improve and expand the documentation to help new users and contributors.
+
+### Guidelines
+
+- Follow the existing code style and structure.
+- Ensure that all tests pass before submitting a pull request.
+- Provide clear and concise commit messages.
+- Open an issue to discuss potential changes before submitting significant modifications.

testing.py

@@ -6,49 +6,64 @@
 from testing.test_runner import TestRunner
 from testing.test_suites_extractor import TestSuitesExtractor
 
+def parse_arguments():
+    ''' Parse the command line arguments '''
+    parser = argparse.ArgumentParser(description="Run test suites in specified environments.")
+    parser.add_argument("-f", "--file", required=True, help="The test suite file name.")
+    parser.add_argument("-V", "--verbose", action="store_true", help="Increase output verbosity.")
+    parser.add_argument("-e", "--envs", nargs="+", default=["cli"], help="Environments to run the tests on (cli, faas).")
+    return parser.parse_args()
 
-def main():
-
-    # Parse the command line arguments
-    parser = argparse.ArgumentParser()
-    parser.add_argument("-f", "--file", action="store", help="the test suite file name")
-    parser.add_argument("-V", "--verbose", action="store_true", help="increase output verbosity")
-    parser.add_argument("-e", "--envs", nargs="+", default=["cli"], help="the environments to run the tests on (cli, faas)")
-    args = parser.parse_args()
-
-    # Set the logger level
+def setup_logger(verbose):
+    ''' Setup logger with the appropriate logging level '''
     logger = Logger.get_instance()
-    logger.set_level("DEBUG" if args.verbose else "INFO")
+    logger.set_level("DEBUG" if verbose else "INFO")
+    return logger
 
-    # Extract the test suites from the test suite file
+def extract_test_suites(file_name):
+    ''' Extract test suites from the test suite file '''
     try:
-        test_suite_file_name = args.file
-        test_suites_extractor = TestSuitesExtractor(test_suite_file_name)
-        _, project_path, repo_url, test_suites = test_suites_extractor.extract_test_suites()
-        project_name = project_path.rsplit('/', maxsplit=1)[-1]
-        logger.info(f"Testing Project: {project_name}")
+        test_suites_extractor = TestSuitesExtractor(file_name)
+        return test_suites_extractor.extract_test_suites()
     except Exception as e:
-        logger.error(f"Error: {e}")
-        exit(1)
-                  
-    # Clone the repo if not already cloned
+        raise RuntimeError(f"Error extracting test suites: {e}")
+
+def clone_repo_if_needed(repo_url):
+    ''' Clone the repository if not already cloned '''
     try:
-        repo_manager = RepoManager(repo_url)
+        repo_manager = RepoManager.get_instance(repo_url)
         repo_manager.clone_repo_if_not_exist()
     except Exception as e:
-        logger.error(f"Error: {e}")
-        exit(1)
+        raise RuntimeError(f"Error cloning repository: {e}")
+
+def deploy_faas_if_needed(envs, project_path, logger):
+    ''' Deploy the project as a local FaaS if required '''
+    if "faas" in envs:
+        deploy_manager = DeployManager.get_instance(project_path)
+        if not deploy_manager.deploy_local_faas():
+            logger.error("Error deploying the project. Removing 'faas' from environments.")
+            envs.remove("faas")
 
-    # Deploy the project as a local faas
-    if "faas" in args.envs:
-        deploy_manager = DeployManager(project_path)
-        if deploy_manager.deploy_local_faas() is False:
-            logger.error("Error deploying the project, remove faas from the envs")
-            args.envs.remove("faas")
-
-    # Run the tests
-    test_runner = TestRunner(args.envs)
+def run_tests(envs, test_suites):
+    ''' Run the tests in the specified environments '''
+    test_runner = TestRunner(envs)
     test_runner.run_tests(test_suites)
 
+def main():
+    args = parse_arguments()
+    logger = setup_logger(args.verbose)
+
+    try:
+        project_name, project_path, repo_url, test_suites = extract_test_suites(args.file)
+        logger.info(f"Testing Project: {project_name}")
+
+        clone_repo_if_needed(repo_url)
+        deploy_faas_if_needed(args.envs, project_path, logger)
+        run_tests(args.envs, test_suites)
+
+    except RuntimeError as e:
+        logger.error(e)
+        exit(1)
+
 if __name__ == "__main__":
     main()

testing/deploy_manager.py

@@ -9,53 +9,68 @@ class DeployManager:
     def __init__(self, project_path):
         if DeployManager._instance is not None:
             raise Exception("This class is a singleton!")
-        else:
-            DeployManager._instance = self
+        DeployManager._instance = self
+
         self.logger = Logger.get_instance()
         self.project_path = project_path
-        self.project_name = project_path.rsplit('/', maxsplit=1)[-1]
-    
+        self.project_name = os.path.basename(project_path)
+
     @staticmethod
-    def get_instance():
+    def get_instance(project_path=None):
         ''' Static access method for singleton '''
         if DeployManager._instance is None:
-            DeployManager(None)
+            if project_path is None:
+                raise ValueError("Project path must be provided for the first instance.")
+            DeployManager(project_path)
         return DeployManager._instance
-
-    def deploy_local_faas(self):
-        ''' Deploy the project as a local faas '''
-        # Set the environment variables
+
+    def set_environment_variables(self, env_vars):
+        ''' Set environment variables '''
         try:
-            os.environ['NODE_ENV'] = 'testing'
-            os.environ['METACALL_DEPLOY_INTERACTIVE'] = 'false'
-        except subprocess.CalledProcessError as e:
-            self.logger.error(f"Error setting the environment variables: {e}")
+            for key, value in env_vars.items():
+                os.environ[key] = value
+        except Exception as e:
+            self.logger.error(f"Error setting environment variables: {e}")
             return False
-        # Deploy the project
+        return True
+
+    def deploy_local_faas(self):
+        ''' Deploy the project as a local FaaS '''
+        env_vars = {
+            'NODE_ENV': 'testing',
+            'METACALL_DEPLOY_INTERACTIVE': 'false'
+        }
+
+        if not self.set_environment_variables(env_vars):
+            return False
+
         try:
             deploy_command = f"metacall-deploy --dev --workdir {self.project_path}"
-            _ = subprocess.run(deploy_command, capture_output=True, text=True, shell=True, check=False)
+            subprocess.run(deploy_command, capture_output=True, text=True, shell=True, check=True)
+            self.logger.debug("Local FaaS deployed successfully.")
             return True
         except subprocess.CalledProcessError as e:
             self.logger.error(f"Error deploying the project: {e}")
             return False
 
     def get_local_base_url(self):
-        ''' Get the base url of the deployed local faas '''
+        ''' Get the base URL of the deployed local FaaS '''
         inspection_command = "metacall-deploy --inspect OpenAPIv3 --dev"
-        # Inspect the deployed project
-        result = subprocess.run(inspection_command, capture_output=True, text=True, shell=True, check=False)
-        # Parse the JSON output to get the server URL and paths
         try:
+            result = subprocess.run(inspection_command, capture_output=True, text=True, shell=True, check=True)
             server_url = json.loads(result.stdout)[0]['servers'][0]['url']
-        except json.JSONDecodeError:
-            self.logger.error(f"Error parsing JSON output: {result.stderr}")
-            return None
-        self.logger.debug(f"Local faas base url: {server_url}")
-        return server_url # e.g. http://localhost:9000/aee940974fd5/examples-testing/v1
+            self.logger.debug(f"Local FaaS base URL: {server_url}")
+            return server_url
+        except subprocess.CalledProcessError as e:
+            self.logger.error(f"Error inspecting the deployed project: {e}")
+        except (json.JSONDecodeError, KeyError) as e:
+            self.logger.error(f"Error parsing JSON output: {e}")
+        return None
 
     def deploy_remote_faas(self):
-        ''' Deploy the project as a remote faas '''
+        ''' Deploy the project as a remote FaaS '''
+        pass
+
     def get_remote_base_url(self):
         ''' Get the base url of the deployed remote faas '''