Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

One-exchange neighborhood may raise ValueError non-deterministically #194

Open
filipbartek opened this issue Sep 15, 2021 · 7 comments · Fixed by #195
Open

One-exchange neighborhood may raise ValueError non-deterministically #194

filipbartek opened this issue Sep 15, 2021 · 7 comments · Fixed by #195

Comments

@filipbartek
Copy link
Contributor

Description

ConfigSpace.util.get_one_exchange_neighbourhood uses np.random to decide whether a new configuration is to be validated (see util.pyx:218). Using np.random for this decision makes the behavior of the function non-deterministic.

Solution proposal

Use the randomness generator random instantiated at util.pyx:130 instead of np.random.

@mfeurer
Copy link
Contributor

mfeurer commented Sep 15, 2021

Thanks a lot for reporting this. Would you mind creating a pull request fixing this? What worries me a bit more, though, is that this is actually triggered - this should not happen at all as the neighborhood generation should only return valid configurations. Do you have a minimal example that reproduces this issue?

@filipbartek
Copy link
Contributor Author

filipbartek commented Sep 15, 2021

Minimal example

Try to run the following PCS with test_sample_configuration_spaces.py: vampire.zip

If all the calls to cs._check_configuration_rigorous are removed from the test body, a ValueError is raised from n.is_valid_configuration() at "test_sample_configuration_spaces.py:74":

E   ValueError: Active hyperparameter 'age_weight_ratio:log_ratio' not specified!

If n.is_valid_configuration() is removed from the test body as well, a ValueError is raised non-deterministically from enumerate(neighborhood) (line 73).

Note that if the calls to cs._check_configuration_rigorous are left in place, a different ValueError is raised from cs._check_configuration_rigorous(default) at "test_sample_configuration_spaces.py:52":

E   ValueError: Inactive hyperparameter 'age_weight_ratio:log_ratio' must not be specified, but has the vector value: '0.7692307692307693'.

@mfeurer
Copy link
Contributor

mfeurer commented Sep 16, 2021

Thanks a lot for the PR and the additional example. So apparently there is an issue in the neighborhood generation that leads to the creation of invalid configurations. If you are familiar with cython and can have a look in there this would be very much appreciated.

@mfeurer
Copy link
Contributor

mfeurer commented Sep 16, 2021

Re-opening this as #195 only solves half of this issue.

@mfeurer mfeurer reopened this Sep 16, 2021
@filipbartek
Copy link
Contributor Author

filipbartek commented Sep 17, 2021

So apparently there is an issue in the neighborhood generation that leads to the creation of invalid configurations. If you are familiar with cython and can have a look in there this would be very much appreciated.

I am investigating the issue. It seems that ConfigSpace.c_util.change_hp_value invoked from get_one_exchange_neighbourhood may return an invalid configuration if a hyperparameter is guarded by a condition of type OrConjunction. The docstring of change_hp_value suggests that the function may return an invalid configuration:

Does not check if the new value is legal. Activates and deactivates other
hyperparameters if necessary. Does not check if new hyperparameter value
results in the violation of any forbidden clauses.

@mfeurer, should change_hp_value in fact guarantee that the output configuration is valid, or is checking the validity a responsibility of the caller (get_one_exchange_neighbourhood)?

@filipbartek
Copy link
Contributor Author

#197 modifies change_hp_value so that it produces a valid configuration in each of the calls invoked by the minimal example described above. Note that the proposed solution probably degrades performance in some cases.

@eddiebergman
Copy link
Contributor

I can confirm that this issue still exists in 1.0.0 and is likely related to the || condition above.

I've created an easier repro example:

repro = r"""
age_weight_ratio:log_ratio real [-10.0, 3.0] [0.0]
saturation_algorithm categorical {discount, fmb, inst_gen, lrs, otter, z3} [lrs]
inst_gen_with_resolution categorical {off, on} [off]

age_weight_ratio:log_ratio | saturation_algorithm != inst_gen || inst_gen_with_resolution == on
inst_gen_with_resolution | saturation_algorithm == inst_gen
""".splitlines()

from ConfigSpace.read_and_write.pcs_new import read

cs = read(repro)
default = cs.get_default_configuration()
print(default)
cs._check_configuration_rigorous(default)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants