-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
API Design Update Including YAMLs #18
Conversation
In the latest push:
|
Latest push:
|
neps.run( | ||
run_pipeline=run_pipeline, | ||
pipeline_space=pipeline_space_fidelity_priors, | ||
root_directory="priorband_bo_user_decided", | ||
max_evaluations_total=1, | ||
searcher="priorband_bo", | ||
initial_design_size=5, | ||
eta=3, | ||
) | ||
|
||
# Testing neps decision tree on deciding the searcher and rejecting the | ||
# additional arguments. | ||
|
||
# Case 1: Choosing priorband | ||
neps.run( | ||
run_pipeline=run_pipeline, | ||
pipeline_space=pipeline_space_fidelity_priors, | ||
root_directory="priorband_neps_decided", | ||
max_evaluations_total=1, | ||
initial_design_size=5, | ||
eta=3, | ||
) | ||
|
||
# Case 2: Choosing bayesian_optimization | ||
neps.run( | ||
run_pipeline=run_pipeline, | ||
pipeline_space=pipeline_space_not_fidelity, | ||
root_directory="bo_neps_decided", | ||
max_evaluations_total=1, | ||
) | ||
|
||
# Case 3: Choosing pibo | ||
neps.run( | ||
run_pipeline=run_pipeline, | ||
pipeline_space=pipeline_space_not_fidelity_priors, | ||
root_directory="pibo_neps_decided", | ||
max_evaluations_total=1, | ||
initial_design_size=5, | ||
) | ||
|
||
# Case 4: Choosing hyperband | ||
neps.run( | ||
run_pipeline=run_pipeline, | ||
pipeline_space=pipeline_space_fidelity, | ||
root_directory="hyperband_neps_decided", | ||
max_evaluations_total=1, | ||
eta=2, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have a neps utility that given a folder can return a status whether runs have started, failed, ended, etc.? Maybe that is for a future PR.
Nevertheless, can we assert and check somehow whether the runs were successful?
Asserting if the kwargs were correctly updated or the right algorithm was called. Simple, sanity checks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Working on status thing for a future push as we discussed.
Asserting is now included in the metahyper, checks the algorithm and simply if any kwargs were updates.
API Design Changes:
neps.run
.searcher_path
inneps.run
.kwargs
: Users can still addkwargs
toneps.run
, which take precedence over YAML configurations.kwargs
: Changingkwargs
is allowed only when the user specifies a searcher.Extras:
__init__
having SearcherMapping: Except forcost_cooling
searcher.SearchConfigs
Class: Provides insights about default NEPS searchers (algorithm, searcher names, kwargs...).Minor Issue:
sample_default_first
was intended to beTrue
when using priors, but it failed when havingneps.FunctionParameter
in the search space. It should now be set manually.