-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
46 additions
and
49 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,97 +1,94 @@ | ||
| # Roadmap | ||
|
|
||
| ## Before 0.12.0 | ||
| ## Before 0.14.0 | ||
|
|
||
| ### Features | ||
|
|
||
| - Allow yaml based input of search space and the target function source to `neps.run` | ||
| - ? Allow yaml based input of search space and the target function source to `neps.run` | ||
| - Generate plot after each evaluation | ||
| - Support conditionals in ConfigSpace search space | ||
| - Support logging of optimizer state details | ||
| - ? Support conditionals in ConfigSpace search space | ||
| - ? Support logging of optimizer state details | ||
| - Nice way to handle Multi-fidelity large scale (slurm script modification) | ||
| - Seamless ddp via cli launcher | ||
| - ? Log priors include again | ||
|
|
||
| ### Fixes | ||
|
|
||
| - Open never closes (talk to Nils) | ||
| - Deadlock in ASHA-like optimizers (talk to Neeratyoy) | ||
| - Extra metahyper sample in [issue 42](https://github.com/automl/neps/issues/42) | ||
| - Tighter type check in search space | ||
| - Unify MFObservedData class for both Hyperband-like fidelities and one-step fidelities | ||
|
|
||
| ### Documentation | ||
|
|
||
| - Improved documentation on all basic usage | ||
| - Improved README on github that links to the documentations | ||
| - Adequate examples targeting different user groups | ||
| - acq search mutation only mutates 1 parameter? | ||
| - Optimize dependencies | ||
| - Double log on neps output | ||
| - Are continuations of errored-out runs removed already? | ||
|
|
||
| ### Refactoring | ||
|
|
||
| - Merge GP and hierarchical GP | ||
| - Merge gpytorch branch | ||
| - ? Merge GP and hierarchical GP | ||
| - Branch cleanup | ||
| - Rethink summary/status API | ||
| - Improve placement of \_post_evaluation_hook_function | ||
| - maintained vs unmaintained optimizers | ||
| - Read and sample at the same time metahyper | ||
| - Metahyper into neps | ||
| - Renamings | ||
| - run_pipeline = evaluate_pipeline | evaluate_pipeline_error | compute_pipeline_error | train_and_evaluate | ||
| - loss = validation_error | error | pipeline_error | ||
| - XParameter = XSpace | ||
| - Rename default-x to prior-x | ||
| - Use max_cost_total everywhere instead of budget | ||
| - ? Renamings | ||
| - run_pipeline = evaluate_pipeline | evaluate_pipeline_error | compute_pipeline_error | train_and_evaluate | ||
| - loss = validation_error | error | pipeline_error | ||
| - XParameter = XSpace | ||
| - Rename default-x to prior-x | ||
| - Use max_cost_total everywhere instead of budget | ||
|
|
||
| ### Tests and tooling | ||
| ### Documentation | ||
|
|
||
| - Keep a changelog | ||
| - Keep citations doc up to date | ||
| - Role of analysing runs needs to be higher in docs | ||
| - Explain what optimizers are run per default / papers higher in docs | ||
| - Hier NAS documentation | ||
| - Rework readme | ||
| - for keyfeatures, 3+4? Mention prior + multi fidelity (cheap approx) | ||
| - Code example of readme should work when copied | ||
| - ? Sync readme with docs landingpage more nicely | ||
| - Parallelization link is broken | ||
|
|
||
| - Add priorband to experimental | ||
| - Add simple regression tests to run on each push | ||
|
|
||
| ## Before 1.0.0 version | ||
|
|
||
| ### Features | ||
|
|
||
| - Seamless ddp via cli launcher | ||
| - Finegrained control over HP user prior | ||
| - Top vs all vs bottom distribution plots | ||
| - Tensorboard visualizations (incumbent plot, ..) | ||
| - Loss distribution plot | ||
| - ? Top vs all vs bottom distribution plots | ||
| - ? Tensorboard visualizations (incumbent plot, ..) | ||
| - ? Loss distribution plot | ||
| - Print search space upon run | ||
| - Add comprehensive regression tests to run manually on the cluster on each version release | ||
| - Utility to generate code for best architecture | ||
| - Core Feature set in terms of research | ||
| - Modular plug-and-play of BoTorch acquisition functions | ||
| - Exploring gradient-based HPO in the NePS framework | ||
| - ? Modular plug-and-play of BoTorch acquisition functions | ||
| - Generate analysis pdf? | ||
|
|
||
| ### Fixes | ||
|
|
||
| - Printing architecture search spaces / search spaces in general | ||
| - Metahyper Refine jobtimelimit feature | ||
| - Optimize dependencies | ||
|
|
||
| ### Refactoring | ||
|
|
||
| - Clean up search spaces classes, unused methods | ||
| - Break up search space and config aspect | ||
| - Remove hnas branch | ||
| - Refactor of constraint grammar | ||
|
|
||
| ### Documentation | ||
| ### Tests and tooling | ||
|
|
||
| - Keep a changelog | ||
| - Add comprehensive regression tests to run manually on the cluster on each version release | ||
| - Regression tests to run on each push | ||
| - ? mdformat for readme | ||
| - ? Darglint | ||
|
|
||
| ## Later version | ||
|
|
||
| ### Documentation | ||
|
|
||
| - ? Doing research with NePS / Documentation on that or full setup | ||
|
|
||
| ### Features | ||
|
|
||
| - neps_examples callable for options of examples | ||
| - Optional argparse adder like pytorch lightning | ||
| - Utility neps.clean to manage existing run results | ||
| - Collect data optionally via phone-home to webserver | ||
| - Add Info dict to status | ||
| - Seed (setting context manager?) | ||
| - BO improvements via Hebo tricks + Mll replacement | ||
| - Checkout Rich logging | ||
|
|
||
| ### Miscellaneous | ||
|
|
||
| - User Mattermost Channel | ||
| - Twitter handle and domain, e.g., neural-pipeline.search | ||
| - Doing research with NePS / Documentation on that or full setup | ||
|
|