Usability and other improvements (#11)

Author: GJFR

.dockerignore

@@ -11,10 +11,12 @@
 aws
 binaries
 browser-repos
+config
 database
 Dockerfile
 docker-compose.yaml
 drivers
+experiments
 extensions
 logs
 support

.env

@@ -0,0 +1,8 @@
+# DO NOT CHANGE OR REMOVE THIS FILE!
+# If you wish to modify a variable, do so by assigning it before the command:
+# For example, `BUGHOG_VERSION=dev docker compose up` to pull the development version.
+
+BUGHOG_VERSION=latest
+BUGHOG_WEB_VERSION=latest
+
+COMPOSE_PROFILES=prod

.gitignore

@@ -1,6 +1,5 @@
 test/resources/repositories
 browser/binaries/
-config/
 database/data/
 !**/.gitkeep
 **/node_modules
@@ -116,7 +115,7 @@ celerybeat.pid
 *.sage.py
 
 # Environments
-.env
+config/.env
 .venv
 env/
 venv/

README.md

@@ -23,83 +23,85 @@ This framework has been developed as part of the _"A Bug's Life: Analyzing the L
     width="100"/>
 
 
-## Usage
-
-### Prerequisites
-- Docker (tested with version 24.0.1)
+## Installation
 
-### Installing
-BugHog can be installed by following the steps below:
+BugHog is compatible with UNIX systems running Docker, including WSL on Windows.
+Follow these steps to get started:
 
-1. Clone this repository:
+1. **Clone this repository:**
 
     ```bash
     git clone https://github.com/DistriNet/BugHog
     cd BugHog
     ```
 
-2. Obtain images:
+2. **Obtain images:**
 
     You will need at least 5 GB of disk space.
+    There are two options available to obtain the BugHog images, and you can switch between them by executing the appropriate command.
 
-    *Option A: Pulling (fastest)*
+    ***Option A:** Pulling (fastest)*
 
     Use the following command to pull the necessary Docker images:
     ```bash
     docker compose pull core worker web
     ```
 
-    *Option B: Building*
+    > :bulb: If you prefer to use a version other than the latest, simply modify the `BUGHOG_VERSION` and / or `BUGHOG_WEB_VERSION` variables accordingly.
+
+    ***Option B:** Building*
 
-    If you want to modify the source code, use the following commands to build the necessary Docker images.
-    Run this script again if you make changes to the source code.
+    Use the following commands to build the Docker images yourself, for instance after you made changes to the source code:
     ```bash
     docker compose build core worker web
     ```
 
     > :bulb: For reference, building takes about 4 minutes on a machine with 8 CPU cores and 8 GB of RAM.
 
-### Starting
-After pulling or building the images, start BugHog with the following command.
-If you switch between pulled and built images, make sure to execute the appropriate commands mentioned above before starting.
+
+## Usage
+
+Launch BugHog using the following command:
 ```bash
-docker compose up core web
+docker compose up
 ```
 
+> :warning: If you use `sudo` with this command, the `PWD` environment variable won't be passed to the BugHog containers, which is necessary for dynamically starting worker containers.
+> To avoid this, explicitly pass on this variable: `sudo PWD=$PWD docker compose up`.
+
 Open your web browser and navigate to [http://localhost:5000](http://localhost:5000) to access the graphical interface.
-If you started BugHog on a remote server, replace localhost with its IP address.
+BugHog is started on a remote server, substitute 'localhost' with its IP address.
+
+BugHog can be stopped through:
+```bash
+docker compose down
+```
 
 > :warning: BugHog's own MongoDB instance will persist data within the [database](database) folder.
 > Be sure to back-up accordingly, or use your own MongoDB instance as explained below.
 
-#### Optional: Use your own MongoDB instance
 
-By default, BugHog uses a MongoDB container.
-you might want to prefer all data to be stored on your own MongoDB instance.
+### Optional: Use your own MongoDB instance
+
+By default, BugHog uses a MongoDB container to store data.
 If you prefer storing data in your own MongoDB instance, follow these steps:
 
-1. Create a `.env` file from `.env.example` (both in the [config](config) folder) and fill in the missing values.
+1. Create a `.env` file from `.env.example` (both in the [config](config) folder) and fill in the missing database values.
 
 2. (Re)start BugHog.
 
-### Stopping
-To stop BugHog, run the following command:
-
-```bash
-docker compose down
-```
 
-### Adding Your Own Experiments
+### Adding your new experiments
 
 Instructions to add your own custom experiments to the server can be found [here](https://github.com/DistriNet/BugHog-web/blob/main/experiments/README.md).
 Be sure to restart the BugHog framework when you add a new experiment:
 
 ```bash
 docker compose down
-docker compose up core web
+docker compose up
 ```
 
-### Development
+## Development
 
 For extending or debugging the Vue UI, the most convenient approach is to launch an interactive Node environment.
 The UI can be visited at [http://localhost:5173](http://localhost:5173).
@@ -108,9 +110,18 @@ The UI can be visited at [http://localhost:5173](http://localhost:5173).
 docker compose up node_dev
 ```
 
-For debugging the core application, consider using the VS Code dev container for an effortless debugging experience.
+For debugging the core application, consider using the VS Code dev container.
 You can utilize the configuration in [.devcontainer](.devcontainer) for this.
 
+
 ## Additional help
 
 Don't hesitate to open a [GitHub issue](https://github.com/DistriNet/BugHog/issues/new) if you come across a bug, want to suggest a feature, or have any questions!
+
+
+## Troubleshooting
+
+### WSL on Windows
+
+- Ensure you clone the BugHog project to the WSL file system instead of the Windows file system, and launch it from there.
+Virtualization between these file systems can cause complications with file management.

analysis/plot_factory.py

@@ -1,16 +1,19 @@
+from bokeh.core.validation import silence
+from bokeh.core.validation.warnings import PALETTE_LENGTH_FACTORS_MISMATCH
 from bokeh.embed import file_html
 from bokeh.models import BasicTickFormatter, ColumnDataSource, HoverTool
 from bokeh.models.glyphs import Circle
 from bokeh.palettes import Iridescent23
 from bokeh.plotting import figure, output_file, show
 from bokeh.resources import CDN
 from bokeh.transform import factor_cmap
-from dotenv import load_dotenv
 
 from bci.database.mongo.mongodb import MongoDB
 from bci.evaluations.logic import PlotParameters
 
 
+silence(PALETTE_LENGTH_FACTORS_MISMATCH, True)
+
 class PlotFactory:
 
     @staticmethod
@@ -120,19 +123,3 @@ def __add_outcome_info(params: PlotParameters, docs: dict):
                 docs_with_outcome.append(new_doc)
         docs_with_outcome = PlotFactory.__transform_to_bokeh_compatible(docs_with_outcome)
         return docs_with_outcome
-
-
-if __name__ == '__main__':
-    load_dotenv()
-    MongoDB.connect()
-    db = MongoDB.get_instance()
-    params = PlotParameters(
-        'c487155',
-        'c487155',
-        'chromium',
-        'csp_chromium',
-        major_version_range=(40, 105),
-    )
-    PlotFactory.show_plot(params, db)
-    html = PlotFactory.create_html_plot_string(params, db)
-    print(html)

bci/configuration.py

@@ -27,6 +27,15 @@ def get_browser_config_class(browser: str):
             case _:
                 raise ValueError(f'Invalid browser \'{browser}\'')
 
+    @staticmethod
+    def check_required_env_parameters() -> bool:
+        if (host_pwd:=os.getenv('HOST_PWD')) in ['', None]:
+            logger.fatal('The "HOST_PWD" variable is not set. If you\'re using sudo, you might have to pass it explicitly, for example "sudo HOST_PWD=$PWD docker compose up"')
+            return False
+        else:
+            logger.debug(f'HOST_PWD={host_pwd}')
+            return True
+
     @staticmethod
     def initialize_folders():
         for browser in ['chromium', 'firefox']:

bci/database/mongo/container.py

@@ -52,7 +52,7 @@ def __create_new_container(user: str, pw: str, db_name, db_host):
             remove=True,
             labels=['bh_db'],
             volumes=[
-                os.path.join(os.getenv('host_pwd'), 'database/data') + ':/data/db'
+                os.path.join(os.getenv('HOST_PWD'), 'database/data') + ':/data/db'
             ],
             ports={'27017/tcp': 27017},
             environment={

bci/distribution/worker_manager.py

@@ -42,7 +42,6 @@ def __run_container(self, params: WorkerParameters, cb: Callable, blocking_wait=
             time.sleep(5)
         container_id = self.container_id_pool.get()
         container_name = f'bh_worker_{container_id}'
-        command = [params.serialize()]
 
         def start_container_thread():
             try:
@@ -64,7 +63,7 @@ def start_container_thread():
                         logger.info(f'Removing old container \'{container.attrs["Name"]}\' to start new one')
                         container.remove(force=True)
                 self.client.containers.run(
-                    'registry.gitlab.kuleuven.be/distrinet/research/bughog/core/worker:latest',
+                    'bughog/worker:latest',
                     name=container_name,
                     hostname=container_name,
                     shm_size='2gb',
@@ -73,18 +72,18 @@ def start_container_thread():
                     detach=False,
                     remove=True,
                     labels=['bh_worker'],
-                    command=command,
+                    command=[params.serialize()],
                     volumes=[
-                        os.path.join(os.getenv('host_pwd'), 'config') + ':/app/config',
-                        os.path.join(os.getenv('host_pwd'), 'browser/binaries/chromium/artisanal') + ':/app/browser/binaries/chromium/artisanal',
-                        os.path.join(os.getenv('host_pwd'), 'browser/binaries/firefox/artisanal') + ':/app/browser/binaries/firefox/artisanal',
-                        os.path.join(os.getenv('host_pwd'), 'experiments') + ':/app/experiments',
-                        os.path.join(os.getenv('host_pwd'), 'browser/extensions') + ':/app/browser/extensions',
-                        os.path.join(os.getenv('host_pwd'), 'logs') + ':/app/logs',
+                        os.path.join(os.getenv('HOST_PWD'), 'config') + ':/app/config',
+                        os.path.join(os.getenv('HOST_PWD'), 'browser/binaries/chromium/artisanal') + ':/app/browser/binaries/chromium/artisanal',
+                        os.path.join(os.getenv('HOST_PWD'), 'browser/binaries/firefox/artisanal') + ':/app/browser/binaries/firefox/artisanal',
+                        os.path.join(os.getenv('HOST_PWD'), 'experiments') + ':/app/experiments',
+                        os.path.join(os.getenv('HOST_PWD'), 'browser/extensions') + ':/app/browser/extensions',
+                        os.path.join(os.getenv('HOST_PWD'), 'logs') + ':/app/logs',
                         '/dev/shm:/dev/shm',
                     ],
                 )
-                logger.debug(f'Container \'{container_name}\' finished experiments with command \'{command}\'')
+                logger.debug(f'Container \'{container_name}\' finished experiments with parameters \'{repr(params)}\'')
                 cb()
             except docker.errors.APIError:
                 logger.error(f'Could not run container \'{container_name}\' or container was unexpectedly removed', exc_info=True)

bci/evaluations/logic.py

@@ -174,15 +174,24 @@ def create_all_test_params(self) -> list[TestParameters]:
                 self.database_collection)
             for mech_group in self.mech_groups]
 
-    def serialize(self) -> str:
-        return json.dumps({
+    def _to_dict(self):
+        return {
             'browser_configuration': self.browser_configuration.to_dict(),
             'evaluation_configuration': self.evaluation_configuration.to_dict(),
             'state': state_factory.to_dict(self.state),
             'mech_groups': self.mech_groups,
             'database_collection': self.database_collection,
             'database_connection_params': self.database_connection_params.to_dict()
-        })
+        }
+
+    def serialize(self) -> str:
+        return json.dumps(self._to_dict())
+
+    def __repr__(self) -> str:
+        param_dict = self._to_dict()
+        # Mask password
+        param_dict['database_connection_params']['password'] = '*'
+        return json.dumps(param_dict)
 
     @staticmethod
     def deserialize(string: str) -> WorkerParameters:

bci/main.py

@@ -21,7 +21,8 @@ class Main:
     def initialize():
         Main.loggers = Loggers()
         Main.loggers.configure_loggers()
-        Main.master = Master()
+        if Global.check_required_env_parameters():
+            Main.master = Master()
 
     @staticmethod
     def is_ready() -> bool: