Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Design Update Including YAMLs #18

Merged
merged 10 commits into from
Oct 16, 2023
Merged

API Design Update Including YAMLs #18

merged 10 commits into from
Oct 16, 2023

Conversation

TarekAbouChakra
Copy link
Collaborator

API Design Changes:

  • Default Searchers as YAML Files: Users manually select NePS YAML files at neps.run.
  • Automatic Searcher Assignment: Users can skip selection, and NEPS assigns a searcher based on the search space.
  • Custom Searcher YAML Files: Users can create and link their custom searcher YAML files using searcher_path in neps.run.
  • Prioritizing kwargs: Users can still add kwargs to neps.run, which take precedence over YAML configurations.
  • Limitation on Changing kwargs: Changing kwargs is allowed only when the user specifies a searcher.

Extras:

  • Replicated Searchers in __init__ having SearcherMapping: Except for cost_cooling searcher.
  • README File: Added detailed instructions.
  • SearchConfigs Class: Provides insights about default NEPS searchers (algorithm, searcher names, kwargs...).

Minor Issue:

  • Initially, sample_default_first was intended to be True when using priors, but it failed when having neps.FunctionParameter in the search space. It should now be set manually.

@TarekAbouChakra
Copy link
Collaborator Author

In the latest push:

  • Fixed some minor bugs.
  • Added a new 'optimizer_info.yaml' file that saves essential information in the optimization directory such as the optimizer's name, algorithm, whether it's user-defined (a boolean), and whether any arguments were accepted (another boolean).
  • Created test cases to validate optimizer's proper functioning. (TODO: still need to add a test case for when the searcher is designed by the user and a searcher_path is passed)

@TarekAbouChakra
Copy link
Collaborator Author

Latest push:

  • Creating a new marker for the yaml_api test case
  • Testing only done locally for the yaml_api

Comment on lines +35 to +82
neps.run(
run_pipeline=run_pipeline,
pipeline_space=pipeline_space_fidelity_priors,
root_directory="priorband_bo_user_decided",
max_evaluations_total=1,
searcher="priorband_bo",
initial_design_size=5,
eta=3,
)

# Testing neps decision tree on deciding the searcher and rejecting the
# additional arguments.

# Case 1: Choosing priorband
neps.run(
run_pipeline=run_pipeline,
pipeline_space=pipeline_space_fidelity_priors,
root_directory="priorband_neps_decided",
max_evaluations_total=1,
initial_design_size=5,
eta=3,
)

# Case 2: Choosing bayesian_optimization
neps.run(
run_pipeline=run_pipeline,
pipeline_space=pipeline_space_not_fidelity,
root_directory="bo_neps_decided",
max_evaluations_total=1,
)

# Case 3: Choosing pibo
neps.run(
run_pipeline=run_pipeline,
pipeline_space=pipeline_space_not_fidelity_priors,
root_directory="pibo_neps_decided",
max_evaluations_total=1,
initial_design_size=5,
)

# Case 4: Choosing hyperband
neps.run(
run_pipeline=run_pipeline,
pipeline_space=pipeline_space_fidelity,
root_directory="hyperband_neps_decided",
max_evaluations_total=1,
eta=2,
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a neps utility that given a folder can return a status whether runs have started, failed, ended, etc.? Maybe that is for a future PR.

Nevertheless, can we assert and check somehow whether the runs were successful?
Asserting if the kwargs were correctly updated or the right algorithm was called. Simple, sanity checks.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working on status thing for a future push as we discussed.

Asserting is now included in the metahyper, checks the algorithm and simply if any kwargs were updates.

src/neps/utils/common.py Outdated Show resolved Hide resolved
@Neeratyoy Neeratyoy merged commit 7bad4b6 into master Oct 16, 2023
10 checks passed
@TarekAbouChakra TarekAbouChakra deleted the yaml-api branch November 7, 2023 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants