Version 2.0.0a1 (#875)

## Big Changes
* We redesigned the scenario class completely. The scenario is implemented as a dataclass now and holds only environment variables (like limitations or save directory). Everything else was moved to the components directly.
* We removed runtime optimization completely (no adaptive capping or imputing anymore).
* We removed the command-line interface and restructured everything alongside. Since SMAC was building upon the command-line interface (especially in combination with the scenario), it was complicated to understand the behavior or find specific implementations. With the removal, we re-wrote everything in python and re-implemented the feature of using scripts as target functions.
* Introducing trials: Each config/seed/budget/instance calculation is a trial.
* The configuration chooser is integrated into the SMBO object now. Therefore, SMBO finally implements an ask-tell interface now.
* Facades are redesigned so that they accept instantiated components directly. If a component is not passed, a default component is used, which is specified for each facade individually in the form of static methods. You can use those static methods directly to adapt a component to your choice.
* A lot of API changes and renamings (e.g., RandomConfigurationChooser -> RandomDesign, Runhistory2EPM -> RunHistoryEncoder).
* Ambiguous variables are renamed and unified across files.
* Dependencies of modules are reduced drastically.
* We incorporated Pynisher 1.0, which ensures limitations cross-platform.
* We incorporated ConfigSpace 0.6, which simplified our examples.
* Examples and documentation are completely reworked. Examples use the new ConfigSpace, and the documentation is adapted to version 2.0.
* Transparent target function signatures: SMAC checks now explicitly if an argument is available (the required arguments are now specified in the intensifier). If there are more arguments that are not passed by SMAC, a warning is raised.
* Components implement a ``meta`` property now, all of which describe the initial state of SMAC. The facade collects all metadata and saves the initial state of the scenario.
* Improved multi-objective in general: RunHistory (in addition to RunHistoryEncoder) both incorporates the multi-objective algorithm. In other words, if the multi-objective algorithm changes the output, it directly affects the optimization process.
* Configspace is saved in json only
* StatusType is saved as integer and not as dict anymore
* We changed the behavior of continuing a run:
    * SMAC automatically checks if a scenario was saved earlier. If there exists a scenario and the initial state is the same, SMAC automatically loads the previous data. However, continuing from that run is not possible yet.
    * If there was a scenario earlier, but the initial state is different, then the user is asked to overwrite the run or to still continue the run although the state is different (Note that this only can happen if the name specified in the scenario is the same). Alternatively, an `old` to the old run is added (e.g., the name was test, it becomes test-old).
    * The initial state of the SMAC run also specifies the name (if no name in the scenario is specified). If the user changes something in the code base or in the scenario, the name and, therefore, the save location automatically changes.

## New Features
* Added a new termination feature: Use `terminate_cost_threshold` in the scenario to stop the optimization after a configuration was evaluated with a cost lower than the threshold.
* Callbacks are completely redesigned. Added callbacks to the facade are called in different positions in the Bayesian optimization loop.
* The multi-objective algorithm `MeanAggregationStrategy` supports objective weights now.
* RunHistory got more methods like ``get_incumbent`` or ``get_pareto_front``.

## Fixes
* You ever noticed that the third configuration has no origin? It's fixed now.
* We fixed ParEGO (it updates every time training is performed now).

## Optimization Changes
* Changed initial design behavior
    * You can add additional configurations now.
    * ``max_ratio`` will limit both ``n_configs`` and ``n_configs_per_hyperparameter`` but not additional configurations
    * Reduced default ``max_ratio`` to 0.1.

## Code Related
* Converted all unittests to pytests.
* Instances, seeds, and budgets can be set to none now. However, mixing none and non-none will throw an exception.

Co-authored-by: Carolin Benjamins <[email protected]>
Co-authored-by: dengdifan <[email protected]>
Co-authored-by: eddiebergman <[email protected]>
Co-authored-by: timruhkopf <[email protected]>
Co-authored-by: Katharina Eggensperger <[email protected]>

Author: 6 people

.github/workflows/dist.yml

@@ -9,12 +9,14 @@ on:
     branches:
       - main
       - development
+      - development-2.0
 
   # Trigger on a open/push to a PR targeting one of these branches
   pull_request:
     branches:
       - main
       - development
+      - development-2.0
 
 jobs:
   dist:
@@ -27,7 +29,7 @@ jobs:
     - name: Setup Python
       uses: actions/setup-python@v4
       with:
-        python-version: "3.9"
+        python-version: "3.10"
 
     - name: Build dist
       run: |

.github/workflows/docs.yml

@@ -11,12 +11,14 @@ on:
     branches:
       - main
       - development
+      - development-2.0
 
   # Trigger on a open/push to a PR targeting one of these branches
   pull_request:
     branches:
       - main
       - development
+      - development-2.0
 
 env:
   name: SMAC3
@@ -31,7 +33,7 @@ jobs:
     - name: Setup Python
       uses: actions/setup-python@v4
       with:
-        python-version: "3.9"
+        python-version: "3.10"
 
     - name: Install dependencies
       run: |

.github/workflows/examples.yml

@@ -9,12 +9,14 @@ on:
     branches:
       - main
       - development
+      - development-2.0
 
   # When a push occurs on a PR that targets these branches
   pull_request:
     branches:
       - main
       - development
+      - development-2.0
 
 jobs:
   run-all-files:

.github/workflows/pre-commit.yml

@@ -9,12 +9,14 @@ on:
     branches:
       - main
       - development
+      - development-2.0
 
   # Triggers with push to a pr aimed at main
   pull_request:
     branches:
       - main
       - development
+      - development-2.0
 
   schedule:
     # Every day at 7AM UTC
@@ -43,7 +45,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9"]
+        python-version: ["3.8", "3.9", "3.10"]
+        # python-version: ["3.8"]
         os: ["ubuntu-latest"]
 
     steps:
@@ -96,7 +99,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.8", "3.9", "3.10"]
+        # python-version: ["3.8"]
         os: ["ubuntu-latest"]
 
     steps:
@@ -132,7 +136,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.8", "3.9", "3.10"]
+        # python-version: ["3.8"]
         os: ["ubuntu-latest"]
 
     steps:

.github/workflows/pytest.yml

@@ -134,9 +134,13 @@ dmypy.json
 .pyre/
 
 *smac3-output_*
+*smac3_output*
 
 # macOS files
 .DS_Store
 
 # Remove docker files
-docker
+docker
+
+# Others
+src

.gitignore

@@ -1,12 +1,6 @@
-# MANIFEST.in defines what files are included/excluded from a built pacakage
-# that gets installed through PyPI or through `pip install`
-# To modify, check out
-# https://packaging.python.org/en/latest/guides/using-manifest-in/
-
-# Include individual files
 include LICENSE.txt
-include requirements.txt
 include smac/py.typed  # This file is to export types
+include smac/logging.yml
 
 prune tests
 prune examples

MANIFEST.in

@@ -7,27 +7,39 @@
 [![codecov
 Status](https://codecov.io/gh/automl/SMAC3/branch/master/graph/badge.svg)](https://codecov.io/gh/automl/SMAC3)
 
-SMAC is a tool for algorithm configuration to optimize the parameters of
-arbitrary algorithms, including hyperparameter optimization of Machine Learning algorithms. The main core consists of
-Bayesian Optimization in combination with an aggressive racing mechanism to
-efficiently decide which of two configurations performs better.
+SMAC is a tool for algorithm configuration to optimize the parameters of arbitrary algorithms, including hyperparameter 
+optimization of Machine Learning algorithms. The main core consists of Bayesian Optimization in combination with an 
+aggressive racing mechanism to efficiently decide which of two configurations performs better.
 
-SMAC3 is written in Python3 and continuously tested with Python 3.7, 3.8, 3.9, and 3.10. Its Random
+SMAC3 is written in Python3 and continuously tested with Python 3.8, 3.9, and 3.10. Its Random
 Forest is written in C++. In further texts, SMAC is representatively mentioned for SMAC3.
 
 > [Documention](https://automl.github.io/SMAC3)
 
 > [Roadmap](https://github.com/orgs/automl/projects/5/views/2)
 
 
+## Important: Changes in v2.0
+
+With the next big major release of SMAC, we drastically boosted the user experience by improving the APIs and how the 
+pipelining is done (see [changelog](CHANGELOG.md)). However, cleaning the code base, we removed the command-line 
+interface (calling a target function from a script is still supported), runtime optimization, and pSMAC. Also,
+ python 3.7 is not supported anymore. If you depend on those functionalities, we kindly ask you to keep using v1.4.
+
+We are excited to introduce the new major release and look forward to developing new features on the new code base. 
+We hope you enjoy this new user experience as much as we do. 🚀
+
+
 ## Installation
 
-Create a new environment with python 3.9 and make sure swig is installed either on your system or
+This instruction is for the installation on a Linux system, for Windows and Mac and further information see the [documention](https://automl.github.io/SMAC3/main/installation.html).
+
+Create a new environment with python 3.10 and make sure swig is installed either on your system or
 inside the environment. We demonstrate the installation via anaconda in the following:
 
 Create and activate environment:
 ```
-conda create -n SMAC python=3.9
+conda create -n SMAC python=3.10
 conda activate SMAC
 ```
 
@@ -44,56 +56,41 @@ pip install smac
 Or alternatively, clone the environment:
 ```
 git clone https://github.com/automl/SMAC3.git && cd SMAC3
-pip install -r requirements.txt
-pip install .
+pip install -e .[dev]
 ```
 
-We refer to the [documention](https://automl.github.io/SMAC3) for further installation options.
-
 
 ## Minimal Example
 
 ```py
-import numpy as np
-
-from sklearn.ensemble import RandomForestClassifier
-from ConfigSpace import ConfigurationSpace
-from ConfigSpace.hyperparameters import UniformIntegerHyperparameter
-from smac.facade.smac_bb_facade import SMAC4BB
-from smac.scenario.scenario import Scenario
-
-
-X_train, y_train = np.random.randint(2, size=(20, 2)), np.random.randint(2, size=20)
-X_val, y_val = np.random.randint(2, size=(5, 2)), np.random.randint(2, size=5)
+from ConfigSpace import Configuration, ConfigurationSpace
 
+import numpy as np
+from smac import HyperparameterOptimizationFacade, Scenario
+from sklearn import datasets
+from sklearn.svm import SVC
+from sklearn.model_selection import cross_val_score
 
-def train_random_forest(config):
-    model = RandomForestClassifier(max_depth=config["depth"])
-    model.fit(X_train, y_train)
+iris = datasets.load_iris()
 
-    # Define the evaluation metric as return
-    return 1 - model.score(X_val, y_val)
 
+def train(config: Configuration, seed: int = 0) -> float:
+    classifier = SVC(C=config["C"], random_state=seed)
+    scores = cross_val_score(classifier, iris.data, iris.target, cv=5)
+    return 1 - np.mean(scores)
 
-if __name__ == "__main__":
-    # Define your hyperparameters
-    configspace = ConfigurationSpace()
-    configspace.add_hyperparameter(UniformIntegerHyperparameter("depth", 2, 100))
 
-    # Provide meta data for the optimization
-    scenario = Scenario({
-        "run_obj": "quality",  # Optimize quality (alternatively runtime)
-        "runcount-limit": 10,  # Max number of function evaluations (the more the better)
-        "cs": configspace,
-    })
+configspace = ConfigurationSpace({"C": (0.100, 1000.0)})
 
-    smac = SMAC4BB(scenario=scenario, tae_runner=train_random_forest)
-    best_found_config = smac.optimize()
+# Scenario object specifying the optimization environment
+scenario = Scenario(configspace, deterministic=True, n_trials=200)
 
+# Use SMAC to find the best configuration/hyperparameters
+smac = HyperparameterOptimizationFacade(scenario, train)
+incumbent = smac.optimize()
 ```
 
-More examples can be found in the [documention](https://automl.github.io/SMAC3).
-
+More examples can be found in the [documention](https://automl.github.io/SMAC3/main/examples/).
 
 
 ## License
@@ -109,15 +106,18 @@ You should have received a copy of the 3-clause BSD license
 along with this program (see LICENSE file).
 If not, see [here](https://opensource.org/licenses/BSD-3-Clause).
 
+
 ## Miscellaneous
 
 SMAC3 is developed by the [AutoML Groups of the Universities of Hannover and
 Freiburg](http://www.automl.org/).
 
-If you have found a bug, please report to [issues](https://github.com/automl/SMAC3/issues). Moreover, we are appreciating any kind of help.
-Find our guidlines for contributing to this package [here](https://github.com/automl/SMAC3/blob/master/.github/CONTRIBUTING.md).
+If you have found a bug, please report to [issues](https://github.com/automl/SMAC3/issues). Moreover, we are 
+appreciating any kind of help. Find our guidlines for contributing to this package 
+[here](https://github.com/automl/SMAC3/blob/master/.github/CONTRIBUTING.md).
 
-If you use SMAC in one of your research projects, please cite our [JMLR paper](https://jmlr.org/papers/v23/21-0888.html):
+If you use SMAC in one of your research projects, please cite our 
+[JMLR paper](https://jmlr.org/papers/v23/21-0888.html):
 ```
 @article{JMLR:v23:21-0888,
   author  = {Marius Lindauer and Katharina Eggensperger and Matthias Feurer and André Biedenkapp and Difan Deng and Carolin Benjamins and Tim Ruhkopf and René Sass and Frank Hutter},
@@ -131,4 +131,4 @@ If you use SMAC in one of your research projects, please cite our [JMLR paper](h
 }
 ```
 
-Copyright (C) 2016-2022  [AutoML Group](http://www.automl.org/).
+Copyright (C) 2016-2022  [AutoML Group](http://www.automl.org).