Version 1.2 [Multi-Objective]

## Features
* Added multi-objective optimization via Mean-Aggregation or Par-EGO (#817, #818). Both approaches normalize
the costs objective-wise based on all data in the history.

## Major Changes
* Results are instantly saved by default now. That means, runhistory.json is saved every time
a trial is added.
* Determinstic behaviour (defined in scenario) is default now. Calling a function/TAE with the same
seed and configuration is expected to be the same.
* Limit resources behaviour is by default false now. This is particually important because pynisher
does not work on all machines (e.g. Colab, Mac, Windows, ...) properly. 
* Renamed scenario object `save_results_instantly` to `save_instantly`.
* Added `multi_objectives` as scenario argument.
* Expanded `cost_for_crash` for multi-objective support.

## Examples
* Integrated spear_qcp example for commandline.
* Python examples are now executed so that the output in the documentation is shown.
* Added multi-objective example.

## Documentation
* Added runhistory page.

## Workflow Clean-up
* Adds PEP 561 compliance (exports types so other packages can be aware of them).
* Allow manual workflow_dispatch on actions that might require it (can manually trigger them from github UI).
* Prevent the double trigger of actions by making push and pull_request and more strict.
* A push to a pull request should no longer cause double the amount of tests to run (along with the other workflows that had on: [push, pull_request].
* Some general cleanup, giving names to some actions, adding some linebreaks to break up things, ...
* Command-line examples are tested again.
* pytest.yaml:
  * Now scheduled to auto run everyday instead of every week.
  * Clean up the body of the steps and move some things to env var.
  * Scaffold for matrix that includes windows and mac testing (currently excluded, see comments).
  * Includes tests for Python 3.10.
  * Changed the boolean flags in the matrix to just be a categorical, easier to read.

## Minor Changes
* Specified that dask should not cache functions/results (#803) .
* Handles invalid configuration vectors gracefully (#776).
* Specified scenario docs that also SMAC options can be used.
* Docs display init methods now.
* Parameters in the docs are shown first now.
* Successive Halving only warns you once if one worker is used only.
* Statistics are better readable now.
* Sobol sequence does not print warnings anymore.

Co-authored-by: Carolin Benjamins <[email protected]>
Co-authored-by: Difan Deng <[email protected]>
Co-authored-by: Tim Ruhkopf <[email protected]>
Co-authored-by: Filip Bartek <[email protected]>
Co-authored-by: Thomas Holvoet <[email protected]>
Co-authored-by: Marius Lindauer <[email protected]>
Co-authored-by: eddiebergman <[email protected]>

.github/workflows/dist.yml

@@ -1,12 +1,30 @@
 name: dist-check
 
-on: [push, pull_request]
+on:
+  # Manual trigger option in github
+  workflow_dispatch:
+
+  # Trigger on push to these branches
+  push:
+    branches:
+      - master
+      - development
+
+  # Trigger on a open/push to a PR targeting one of these branches
+  pull_request:
+    branches:
+      - master
+      - development
+
+
 
 jobs:
   dist:
     runs-on: ubuntu-latest
+
     steps:
-    - uses: actions/checkout@v2
+    - name: Checkout
+      uses: actions/checkout@v2
 
     - name: Setup Python
       uses: actions/setup-python@v2
@@ -28,9 +46,13 @@ jobs:
       run: |
         last_dist=$(ls -t dist/smac-*.tar.gz | head -n 1)
         pip install $last_dist
-        
+
     - name: PEP 561 Compliance
       run: |
         pip install mypy
+
         cd ..  # required to use the installed version of smac
-        if ! python -c "import smac"; then exit 1; fi
+
+        # Note that doesn't do a mypy check, only that the package exports type
+        # information
+        if ! mypy -c "import smac"; then exit 1; fi

.github/workflows/docs.yml

@@ -1,11 +1,30 @@
-name: Docs
-on: [pull_request, push]
+name: docs
+
+on:
+  # Manual trigger option in github
+  # This won't push to github pages where docs are hosted due
+  # to the gaurded if statement in those steps
+  workflow_dispatch:
+
+  # Trigger on push to these branches
+  push:
+    branches:
+      - master
+      - development
+
+  # Trigger on a open/push to a PR targeting one of these branches
+  pull_request:
+    branches:
+      - master
+      - development
 
 jobs:
   build-and-deploy:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+
+    - name: Checkout
+      uses: actions/checkout@v2
 
     - name: Setup Python
       uses: actions/setup-python@v2

.github/workflows/examples.yml

@@ -1,6 +1,20 @@
-name: Examples
+name: examples
 
-on: [push, pull_request]
+on:
+  # Manual trigger option in github
+  workflow_dispatch:
+
+  # Trigger on push to these branches
+  push:
+    branches:
+      - master
+      - development
+
+  # Trigger on a open/push to a PR targeting one of these branches
+  pull_request:
+    branches:
+      - master
+      - development
 
 jobs:
   ubuntu:
@@ -17,9 +31,37 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: ${{ matrix.python-version }}
+
     - name: Conda Install test dependencies
       run: |
         # Miniconda is available in $CONDA env var
         $CONDA/bin/conda create -n testenv --yes pip wheel gxx_linux-64 gcc_linux-64 swig python=${{ matrix.python-version }}
         $CONDA/envs/testenv/bin/python3 -m pip install --upgrade pip
-        $CONDA/envs/testenv/bin/pip3 install -e .[all]
+        $CONDA/envs/testenv/bin/pip3 install .[all]
+
+    - name: Spear-QCP ROAR (Commandline)
+      timeout-minutes: 20
+      run: |
+        # Activate anaconda so default python is from conda
+        export PATH="$CONDA/envs/testenv/bin:$PATH"
+
+        # cd examples/commandline
+        bash examples/commandline/spear_qcp_roar.sh
+
+    - name: Spear-QCP SMAC (Commandline)
+      timeout-minutes: 20
+      run: |
+        # Activate anaconda so default python is from conda
+        export PATH="$CONDA/envs/testenv/bin:$PATH"
+
+        # cd examples/commandline
+        bash examples/commandline/spear_qcp_smac.sh
+
+    - name:  Branin (Commandline)
+      timeout-minutes: 20
+      run: |
+        # Activate anaconda so default python is from conda
+        export PATH="$CONDA/envs/testenv/bin:$PATH"
+
+        # cd examples/commandline
+        python ./scripts/smac.py --scenario examples/commandline/branin/scenario.txt

.github/workflows/pre-commit.yaml

@@ -1,12 +1,29 @@
 name: pre-commit
 
-on: [push, pull_request]
+on:
+  # Manual trigger option in github
+  workflow_dispatch:
+
+  # Trigger on push to these branches
+  push:
+    branches:
+      - master
+      - development
+
+  # Trigger on a open/push to a PR targeting one of these branches
+  pull_request:
+    branches:
+      - master
+      - development
 
 jobs:
+
   run-all-files:
     runs-on: ubuntu-latest
+
     steps:
-    - uses: actions/checkout@v2
+    - name: Checkout
+      uses: actions/checkout@v2
 
     - name: Setup Python 3.7
       uses: actions/setup-python@v2
@@ -17,7 +34,7 @@ jobs:
       run: |
         pip install pre-commit
         pre-commit install
-        
+
     - name: Run pre-commit
       run: |
         pre-commit run --all-files