update to master

automl · Jul 2, 2024 · 3cca67b · 3cca67b
2 parents ae6a92c + c3dc988
commit 3cca67b
Show file tree

Hide file tree

Showing 12 changed files with 248 additions and 262 deletions.
diff --git a/README.md b/README.md
@@ -5,61 +5,62 @@
 [![License](https://img.shields.io/pypi/l/neural-pipeline-search?color=informational)](LICENSE)
 [![Tests](https://github.com/automl/neps/actions/workflows/tests.yaml/badge.svg)](https://github.com/automl/neps/actions)
 
-Welcome to NePS, a powerful and flexible Python library for hyperparameter optimization (HPO) and neural architecture search (NAS) with its primary goal: enable HPO adoption in practice for deep learners!
+Welcome to NePS, a powerful and flexible Python library for hyperparameter optimization (HPO) and neural architecture search (NAS) with its primary goal: enable HPO and NAS for deep learners!
 
-NePS houses recently published and some more well-established algorithms that are all capable of being run massively parallel on any distributed setup, with tools to analyze runs, restart runs, etc.
-
-Take a look at our [documentation](https://automl.github.io/neps/latest/) and continue following through current README for instructions on how to use NePS!
+NePS houses recently published and also well-established algorithms that can all be run massively parallel on distributed setups, with tools to analyze runs, restart runs, etc., all tailored to the needs of deep learning experts.
 
+Take a look at our [documentation](https://automl.github.io/neps/latest/) for all the details on how to use NePS!
 
 ## Key Features
 
-In addition to the common features offered by traditional HPO and NAS libraries, NePS stands out with the following key features:
+In addition to the features offered by traditional HPO and NAS libraries, NePS, e.g., stands out with:
 
 1. [**Hyperparameter Optimization (HPO) With Prior Knowledge:**](neps_examples/template/priorband_template.py)
-    - NePS excels in efficiently tuning hyperparameters using algorithms that enable users to make use of their prior knowledge within the search space. This is leveraged by the insights presented in:
-        - [PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning](https://arxiv.org/abs/2306.12370)
-        - [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://arxiv.org/abs/2204.11051)
 
-2. [**Neural Architecture Search (NAS) With Context-free Grammar Search Spaces:**](neps_examples/basic_usage/architecture.py)
-    - NePS is equipped to handle context-free grammar search spaces, providing advanced capabilities for designing and optimizing architectures. this is leveraged by the insights presented in:
-        - [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://arxiv.org/abs/2211.01842)
+   - NePS excels in efficiently tuning hyperparameters using algorithms that enable users to make use of their prior knowledge within the search space. This is leveraged by the insights presented in:
+     - [PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning](https://arxiv.org/abs/2306.12370)
+     - [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://arxiv.org/abs/2204.11051)
+
+1. [**Neural Architecture Search (NAS) With Context-free Grammar Search Spaces:**](neps_examples/basic_usage/architecture.py)
+
+   - NePS is equipped to handle context-free grammar search spaces, providing advanced capabilities for designing and optimizing architectures. this is leveraged by the insights presented in:
+     - [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://arxiv.org/abs/2211.01842)
+
+1. [**Easy Parallelization and Resumption of Runs:**](https://automl.github.io/neps/latest/examples/efficiency/)
+
+   - NePS simplifies the process of parallelizing optimization tasks both on individual computers and in distributed
+     computing environments. It also allows users to conveniently resume these optimization tasks after completion to
+     ensure a seamless and efficient workflow for long-running experiments.
 
-3. [**Easy Parallelization and Resumption of Runs:**](docs/parallelization.md)
-      - NePS simplifies the process of parallelizing optimization tasks both on individual computers and in distributed
-   computing environments. It also allows users to conveniently resume these optimization tasks after completion to
-   ensure a seamless and efficient workflow for long-running experiments.
+1. [**Seamless User Code Integration:**](neps_examples/template/)
 
-4. [**Seamless User Code Integration:**](neps_examples/template/)
-    - NePS's modular design ensures flexibility and extensibility. Integrate NePS effortlessly into existing machine learning workflows.
+   - NePS's modular design ensures flexibility and extensibility. Integrate NePS effortlessly into existing machine learning workflows.
 
-## Getting Started
+## Installation
 
-### 1. Installation
-NePS requires Python 3.8 or higher. You can install it via pip or from source.
+To install the latest release from PyPI run
 
-Using pip:
 ```bash
 pip install neural-pipeline-search
 ```
 
-> Note: As indicated with the `v0.x.x` version number, NePS is early stage code and APIs might change in the future.
+To get the latest version from github run
 
-You can install from source by cloning the repository and running:
 ```bash
-git clone [email protected]:automl/neps.git
-cd neps
-poetry install
+pip install git+https://github.com/automl/neps.git
 ```
 
-### 2. Basic Usage
+> Note: As indicated with the `v0.x.x` version number APIs will change in the future.
+
+
+## Basic Usage
 
 Using `neps` always follows the same pattern:
 
 1. Define a `run_pipeline` function capable of evaluating different architectural and/or hyperparameter configurations
    for your problem.
-2. Define a search space named `pipeline_space` of those Parameters e.g. via a dictionary
-3. Call `neps.run` to optimize `run_pipeline` over `pipeline_space`
+1. Define a search space named `pipeline_space` of those Parameters e.g. via a dictionary
+1. Call `neps.run` to optimize `run_pipeline` over `pipeline_space`
 
 In code, the usage pattern can look like this:
 
@@ -89,66 +90,44 @@ def run_pipeline(
     }
 
 
-# 2. Define a search space of parameters; use the same names for the parameters as in run_pipeline
+# 2. Define a search space of parameters; use the same parameter names as in run_pipeline
 pipeline_space = dict(
-    hyperparameter_b=neps.IntegerParameter(
-        lower=1, upper=42, is_fidelity=True
-    ),  # Mark 'is_fidelity' as true for a multi-fidelity approach.
     hyperparameter_a=neps.FloatParameter(
-        lower=0.001, upper=0.1, log=True
-    ),  # If True, the search space is sampled in log space.
-    architecture_parameter=neps.CategoricalParameter(
-        ["option_a", "option_b", "option_c"]
+        lower=0.001, upper=0.1, log=True  # The search space is sampled in log space
     ),
+    hyperparameter_b=neps.IntegerParameter(lower=1, upper=42),
+    architecture_parameter=neps.CategoricalParameter(["option_a", "option_b"]),
 )
 
-if __name__ == "__main__":
-    # 3. Run the NePS optimization
-    logging.basicConfig(level=logging.INFO)
-    neps.run(
-        run_pipeline=run_pipeline,
-        pipeline_space=pipeline_space,
-        root_directory="path/to/save/results",  # Replace with the actual path.
-        max_evaluations_total=100,
-        searcher="hyperband"  # Optional specifies the search strategy,
-        # otherwise NePs decides based on your data.
-    )
+
+# 3. Run the NePS optimization
+logging.basicConfig(level=logging.INFO)
+neps.run(
+    run_pipeline=run_pipeline,
+    pipeline_space=pipeline_space,
+    root_directory="path/to/save/results",  # Replace with the actual path.
+    max_evaluations_total=100,
+)
 ```
 
 ## Examples
 
 Discover how NePS works through these practical examples:
-* **[Pipeline Space via YAML](neps_examples/basic_usage/hpo_usage_example.py)**: Explore how to define the `pipeline_space` using a
-  YAML file instead of a dictionary.
-
-* **[Hyperparameter Optimization (HPO)](neps_examples/basic_usage/hyperparameters.py)**: Learn the essentials of hyperparameter optimization with NePS.
-
-* **[Architecture Search with Primitives](neps_examples/basic_usage/architecture.py)**: Dive into architecture search using primitives in NePS.
-
-* **[Multi-Fidelity Optimization](neps_examples/efficiency/multi_fidelity.py)**: Understand how to leverage multi-fidelity optimization for efficient model tuning.
 
-* **[Utilizing Expert Priors for Hyperparameters](neps_examples/efficiency/expert_priors_for_hyperparameters.py)**: Learn how to incorporate expert priors for more efficient hyperparameter selection.
+- **[Hyperparameter Optimization (HPO)](neps_examples/basic_usage/hyperparameters.py)**: Learn the essentials of hyperparameter optimization with NePS.
 
-* **[Additional NePS Examples](neps_examples/)**: Explore more examples, including various use cases and advanced configurations in NePS.
+- **[Architecture Search with Primitives](neps_examples/basic_usage/architecture.py)**: Dive into architecture search using primitives in NePS.
 
-## Documentation
+- **[Multi-Fidelity Optimization](neps_examples/efficiency/multi_fidelity.py)**: Understand how to leverage multi-fidelity optimization for efficient model tuning.
 
-For more details and features please have a look at our [documentation](https://automl.github.io/neps/latest/)
+- **[Utilizing Expert Priors for Hyperparameters](neps_examples/efficiency/expert_priors_for_hyperparameters.py)**: Learn how to incorporate expert priors for more efficient hyperparameter selection.
 
-## Analysing runs
-
-See our [documentation on analysing runs](https://automl.github.io/neps/latest/analyse).
+- **[Additional NePS Examples](neps_examples/)**: Explore more examples, including various use cases and advanced configurations in NePS.
 
 ## Contributing
 
-Please see the [documentation for contributors](https://automl.github.io/neps/latest/contributing/).
+Please see the [documentation for contributors](https://automl.github.io/neps/latest/dev_docs/contributing/).
 
 ## Citations
 
-Please consider citing us if you use our tool!
-
-Refer to our [documentation on citations](https://automl.github.io/neps/latest/citations/).
-
-## Alternatives
-
-NePS does not cover your use-case? Have a look at [some alternatives](https://automl.github.io/neps/latest/alternatives).
+For pointers on citing the NePS package and papers refer to our [documentation on citations](https://automl.github.io/neps/latest/citations/).
diff --git a/docs/dev_docs/roadmap.md b/docs/dev_docs/roadmap.md
@@ -1,97 +1,87 @@
 # Roadmap
 
-## Before 0.12.0
+## Next up
 
 ### Features
 
-- Allow yaml based input of search space and the target function source to `neps.run`
-- Generate plot after each evaluation
-- Support conditionals in ConfigSpace search space
-- Support logging of optimizer state details
+- Improve handling of multi-fidelity for large scale (slurm script modification)
+- Evaluate and maybe improve ease-of-use of NePS and DDP etc.
+- Optimize dependencies
 
 ### Fixes
 
-- Open never closes (talk to Nils)
-- Deadlock in ASHA-like optimizers (talk to Neeratyoy)
-- Extra metahyper sample in [issue 42](https://github.com/automl/neps/issues/42)
-- Tighter type check in search space
-- Unify MFObservedData class for both Hyperband-like fidelities and one-step fidelities
+- Acq search mutation for HPs potentially only mutates 1 parameter
+- `ignore_errors` should work seamlessly with all optimizers
+
+### Refactoring
+
+- Rename: run_pipeline = evaluate_pipeline | evaluate_pipeline_error | compute_pipeline_error | train_and_evaluate
+- Rename: loss = validation_error | error | pipeline_error
+- Rename: XParameter = XSpace or just X?
+- Rename: default-x to prior-x
+- Rename: Use max_cost_total everywhere instead of budget
 
 ### Documentation
 
-- Improved documentation on all basic usage
-- Improved README on github that links to the documentations
-- Adequate examples targeting different user groups
+- Keep citations doc up to date
+- Role of analysing runs needs to be higher in docs
+- Explain what optimizers are run per default / papers higher in docs
+- Rework README.md
+    - Rethink key features. Who is reading this?  Mention multi-fidelity / scaling algorithmis?
+    - Code example of readme should work when copied
+    - Keep README synced with docs landingpage more nicely
 
-### Refactoring
+### Tests
+
+- Regression tests to run on each push
 
-- Merge GP and hierarchical GP
-- Merge gpytorch branch
-- Rethink summary/status API
-- Improve placement of \_post_evaluation_hook_function
-- maintained vs unmaintained optimizers
-- Read and sample at the same time metahyper
-- Metahyper into neps
-- Renamings
-  - run_pipeline = evaluate_pipeline | evaluate_pipeline_error | compute_pipeline_error | train_and_evaluate
-  - loss = validation_error | error | pipeline_error
-  - XParameter = XSpace
-  - Rename default-x to prior-x
-  - Use max_cost_total everywhere instead of budget
-
-### Tests and tooling
-
-- Add priorband to experimental
-- Add simple regression tests to run on each push
 
 ## Before 1.0.0 version
 
 ### Features
 
-- Seamless ddp via cli launcher
-- Finegrained control over HP user prior
-- Top vs all vs bottom distribution plots
-- Tensorboard visualizations (incumbent plot, ..)
-- Loss distribution plot
+- Generate plot after each evaluation
+- Finegrained control over user prior
 - Print search space upon run
-- Add comprehensive regression tests to run manually on the cluster on each version release
 - Utility to generate code for best architecture
-- Core Feature set in terms of research
-- Modular plug-and-play of BoTorch acquisition functions
-- Exploring gradient-based HPO in the NePS framework
+- Core algorithmic feature set (research)
 
 ### Fixes
 
-- Printing architecture search spaces / search spaces in general
-- Metahyper Refine jobtimelimit feature
-- Optimize dependencies
+- Contact https://pypi.org/project/neps/ to free up `pip install neps`
 
 ### Refactoring
 
-- Clean up search spaces classes, unused methods
+- Improve neps.optimizers:
+    - Maintained vs unmaintained optimizers
+    - Remove unnecessary / broken optimizers
 - Break up search space and config aspect
-- Remove hnas branch
-- Refactor of constraint grammar
 
 ### Documentation
 
-- Keep a changelog
+- NAS documentation
+
+### Tests
+
+- Regression tests to run on each push
 
-## Later version
+## After 1.0.0
 
 ### Features
 
-- neps_examples callable for options of examples
-- Optional argparse adder like pytorch lightning
 - Utility neps.clean to manage existing run results
 - Collect data optionally via phone-home to webserver
-- Add Info dict to status
-- Seed (setting context manager?)
-- BO improvements via Hebo tricks + Mll replacement
-- Checkout Rich logging
 
-### Miscellaneous
+### Documentation
+
+- Keep a changelog
+
+
+## Rethink
 
-- User Mattermost Channel
-- Twitter handle and domain, e.g., neural-pipeline.search
-- Doing research with NePS / Documentation on that or full setup
+- Log priors include again
+- Allow yaml based input of search space and the target function source to `neps.run`
+- Support conditionals in ConfigSpace search space
+- Support logging of optimizer state details
+- Merge GP and hierarchical GP
+- Generate analysis pdf