Adding FOREM to openctest framework

core/README.md

@@ -14,13 +14,15 @@ A prototype for generating and running ctests. Below are the projects we current
 - Hbase 2.2.2: `hbase-server`.
 - ZooKeeper 3.5.6: `zookeeper-server`.
 - Alluxio 2.1.0: `core`.
+- forem 0.11.1: `forem`.
 
 We also provided our instrumented versions of the above projects:
 
 - Hadoop 2.8.5: https://github.com/xlab-uiuc/hadoop
 - Hbase 2.2.2: https://github.com/xlab-uiuc/hbase
 - ZooKeeper 3.5.6: https://github.com/xlab-uiuc/zookeeper
 - Alluxio 2.1.0: https://github.com/xlab-uiuc/alluxio
+- forem 0.11.1: https://github.com/ishitakarna/forem
 
 Our instrumented version projects have two branches: 
 - `ctest-injection`: branch with "Intercept Configuration API" instrumentation (See `ADDING_NEW_PROJECT.md`). This branch is used by `generate_ctest` and `run_ctest`.
@@ -59,7 +61,7 @@ To generate ctests or run ctest, you need to first clone the target project.
 1. In `openctest/core`, run `./add_project.sh <main project>` to clone the project, switch to and build the branch `ctest-injection`. This branch will be later used by `generate_ctest` and `run_ctest`.
 2. In `openctest/core/identify_param`, run `./add_project.sh <main project>` to clone the project, switch to and build the branch `ctest-logging`. This branch will be later used by `identify_param`.
 
-`<main project>` can be `hadoop`, `hbase`, `zookeeper` or `alluxio`.
+`<main project>` can be `hadoop`, `hbase`, `zookeeper`, `alluxio`, or `forem`.
 
 ## Usage
 

core/add_project.sh

@@ -48,6 +48,13 @@ function setup_alluxio() {
     mvn clean install -DskipTests -Dcheckstyle.skip -Dlicense.skip -Dfindbugs.skip -Dmaven.javadoc.skip=true
 }
 
+function setup_forem() {
+    [ ! -d "app/forem" ] && git clone https://github.com/ishitakarna/forem.git app/forem
+    cd app/forem
+    git fetch && git checkout mapping-patch
+    npm install
+}
+
 function usage() {
     echo "Usage: add_project.sh <main project>"
     exit 1
@@ -64,6 +71,7 @@ function main() {
             hbase) setup_hbase ;;
             zookeeper) setup_zookeeper ;;
             alluxio) setup_alluxio ;;
+            forem) setup_forem ;;
             *) echo "Unexpected project: $project - only support hadoop, hbase, zookeeper and alluxio." ;;
         esac
     fi

core/forem.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+if [ -z "$1" ]
+then
+    echo "Please provide a version number as an argument."
+    exit 1
+fi
+
+VERSION=$1
+
+REPO_URL="https://github.com/forem/forem.git"
+
+git clone $REPO_URL
+cd forem
+git checkout tags/$VERSION -b $VERSION-branch
+
+if ! command -v npm &> /dev/null
+then
+    echo "npm could not be found. Please install it."
+    exit 1
+fi
+
+npm install
+
+bundle install

core/generate_ctest/README.md

@@ -28,6 +28,12 @@ p_input = {
 
 *Second*, run `./generate_ctest.sh`.
 
+For Forem,
+```
+python3 gen_ctest.py <params_file> <test_cases_file> <report_file>
+```
+
+
 ### Result
 
 **Test result** is collected per parameter and stored in `test_result/<project>/test_reuslt_<parameter>.tsv`.  `test_result/<project>/missing_test_<parameter>.tsv` stores tests whose Maven test report was missing while the test result is being collected.

core/generate_ctest/gen_ctest.py

@@ -0,0 +1,79 @@
+import json
+import sys
+import subprocess
+
+
+def load_params(file_path):
+    params = []
+    with open(file_path, 'r') as file:
+        for line in file:
+            parts = line.strip().split('\t')
+            if len(parts) >= 2:
+                params.append((parts[0], parts[1]))  # (parameter, value)
+    return params
+
+
+def load_test_cases(file_path):
+    with open(file_path, 'r') as file:
+        return json.load(file)
+
+
+def run_test_case(param, value, test_case_path, test_case_name):
+    try:
+        # Constructing the command
+        command = ['python3', 'ctest_runner.py', param, value, test_case_path, test_case_name]
+
+
+        print(command)
+
+        # Running the test case
+        result = subprocess.run(command, check=True, capture_output=True, text=True)
+
+
+        print(result.returncode)
+
+
+        # Here you should parse the result to determine pass/fail and execution time
+        # This is dependent on the output format of your testing framework
+        test_result = 'p' if result.returncode == 0 else 'f'
+        test_time = 'unknown'  # Extract this from the result if possible
+
+
+        return test_result, test_time
+    except subprocess.CalledProcessError as e:
+        print(f"Error running the test case: {e}")
+        return 'f', 'unknown'
+
+
+def generate_test_report(params_file, test_cases_file, report_file):
+    params = load_params(params_file)
+    test_cases = load_test_cases(test_cases_file)
+
+
+    with open(report_file, 'w') as file:
+        file.write("parameter\ttest_case\tvalue\ttest_result\ttestcase_time\n")
+
+
+        for param, value in params:
+            if param in test_cases:
+                for test_case in test_cases[param]:
+                    test_case_name = test_case.split(': ')[-1]
+                    test_case_path = test_case.split(':')[0]
+
+
+                    test_result, test_time = run_test_case(param, value, test_case_path, test_case_name)
+                    file.write(f"{param}\t{test_case_name}\t{value}\t{test_result}\t{test_time}\n")
+
+
+if __name__ == "__main__":
+    if len(sys.argv) != 4:
+        print("Usage: gen_ctest.py <params_file> <test_cases_file> <report_file>")
+        sys.exit(1)
+
+
+    params_file = sys.argv[1]
+    test_cases_file = sys.argv[2]
+    report_file = sys.argv[3]
+
+
+    generate_test_report(params_file, test_cases_file, report_file)