Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling min_budget = max_budget situation #33

Closed
Bronzila opened this issue Jun 26, 2023 · 4 comments · Fixed by #46 or #50
Closed

Handling min_budget = max_budget situation #33

Bronzila opened this issue Jun 26, 2023 · 4 comments · Fixed by #46 or #50
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@Bronzila
Copy link
Collaborator

Bronzila commented Jun 26, 2023

When specifying the minimum budget equal to the maximum budget, DEHB performs plain DE. There are two potential ways of dealing with this:

  1. Break and throw an error communicating to the user, that this setup would result in plain DE and therefore it wont be done or
  2. Return a warning and perform DE with a population consisting of one individual on the max_budget. However only random configs are used for mutation

Personally, I would prefer option 2, since I do not feel the need to restrict the user. However communicating, that this setup does not perform DEHB should be done.

Edit: Currently we break and throw an error. However if we skip the assertion it runs through normally --> TODO: Check what happens under the hood

@Bronzila Bronzila added enhancement New feature or request good first issue Good for newcomers labels Jun 26, 2023
@Bronzila
Copy link
Collaborator Author

Bronzila commented Jul 19, 2023

Just checked what happens under the hood:
Essentially it runs DE with a population consisting of one individual on the max_budget. However only random configs are used for mutation, since the parent and global pool are obviously empty, since there are no promotions from lower to higher budgets.
@Neeratyoy I don't think it makes sense to run DEHB in such a setting, but given, that it is possible, should we still allow it? If we allow it, I would obviously add a warning describing what would happen as above. What do you think?

@Neeratyoy
Copy link
Collaborator

Neeratyoy commented Jul 21, 2023

Would be nice to compare what the likes of SMAC or NePS do in such cases and be consistent if possible.

@Bronzila
Copy link
Collaborator Author

When using the SuccessiveHalving and Hyperband intensifiers SMAC actually restricts the user to only use max_budget > min_budget as seen here here.
Should we stay consistent with SMAC or should I still implement the idea of running plain DE?

@Neeratyoy
Copy link
Collaborator

It is fair to throw a warning and halt execution if min and max budgets are the same.
We should undoubtedly give better warnings.
To be fancy, we could even dump a parameterized call to DE and suggest to the user that if they have a fixed fidelity, they can instead run DE.

@Bronzila Bronzila closed this as completed Aug 2, 2023
@Bronzila Bronzila mentioned this issue Aug 2, 2023
@Bronzila Bronzila linked a pull request Aug 2, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
2 participants