Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create easy way to remove hyperparameters #267

Open
eddiebergman opened this issue Aug 19, 2022 · 7 comments · May be fixed by #332
Open

Create easy way to remove hyperparameters #267

eddiebergman opened this issue Aug 19, 2022 · 7 comments · May be fixed by #332

Comments

@eddiebergman
Copy link
Contributor

eddiebergman commented Aug 19, 2022

YAHPOBench has the task id in it's configuration as a hyperparamter which is not something you would like any optimizer to know about. Should be an easy way to remove a hyperparamter by name.

If this violates some conditional where the remove hyperparamter is what is being condition on, i.e. a is active when b is x and we remove b from the space. Then we need to remove this condition too. Likewise for forbidden clauses.

Currently just removing the task id manually and updating the cache as it's not a dependant of anything.

@eddiebergman
Copy link
Contributor Author

eddiebergman commented Aug 19, 2022

Current hack:

def remove_hyperparameter(name: str, space: ConfigurationSpace) -> None:
    """Removes a hyperparameter from a configuration space

    Essentially undoes the operations done by adding a hyperparamter
    and then runs the same validation checks that is done in ConfigSpace

    NOTE
    ----
    * Doesn't account for conditionals

    Parameters
    ----------
    name : str
        The name of the hyperparamter to remove

    space : ConfigurationSpace
        The space to remove it from
    """
    if name not in space._hyperparameters:
        raise ValueError(f"{name} not in {space}")

    assert name not in space._conditionals, "Can't handle conditionals"

    assert not any(
        name != f.hyperparameter.name for f in space.get_forbiddens()
    ), "Can't handle forbiddens"

    # No idea what this is really for
    root = "__HPOlib_configuration_space_root__"

    # Remove it from children
    if root in space._children and name in space._children[root]:
        del space._children[root][name]

    # Remove it from parents
    if root in space._parents and name in space._parents[root]:
        del space._parents[root][name]

    # Remove it from indices
    if name in space._hyperparameter_idx:
        del space._hyperparameter_idx[name]

        # We re-enumerate the dict
        space._hyperparameter_idx = {
            name: idx for idx, name in enumerate(space._hyperparameter_idx)
        }

    # Finally, remove it from the known parameter
    del space._hyperparameters[name]

    # Update according to what adding does `add_hyperparameter()`
    space._update_cache()
    space._check_default_configuration()  # TODO: Get sporadic failures here?
    space._sort_hyperparameters()

    return

@mfeurer
Copy link
Contributor

mfeurer commented Aug 22, 2022

Hey, this would also require to remove it from conditions and forbiddens, or to remove conditions and forbiddens if hyperparameters are in there.

@eddiebergman
Copy link
Contributor Author

True, good spot and thank you!

@eddiebergman
Copy link
Contributor Author

I just added the following two assertions for now until I can figure out how to do it properly:

    assert name not in space._conditionals, "Can't handle conditionals"

    assert not any(
        name != f.hyperparameter.name for f in space.get_forbiddens()
    ), "Can't handle forbiddens"

@eddiebergman
Copy link
Contributor Author

eddiebergman commented Aug 24, 2022

Turns out the above hack is stochastic? The _check_default_configuration seems to randomly fail 1/4 of the time. I just get the error:

mfpbench/yahpo/benchmark.py:94: in __init__
    remove_hyperparameter("OpenML_task_id", space)
mfpbench/util.py:79: in remove_hyperparameter
    space._check_default_configuration()
ConfigSpace/configuration_space.pyx:1068: in ConfigSpace.configuration_space.ConfigurationSpace._check_default_configuration
    ???
ConfigSpace/configuration_space.pyx:1465: in ConfigSpace.configuration_space.Configuration.__init__
    ???
ConfigSpace/configuration_space.pyx:1495: in ConfigSpace.configuration_space.Configuration.is_valid_configuration
    ???
ConfigSpace/c_util.pyx:38: in ConfigSpace.c_util.check_configuration
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   IndexError: index 7 is out of bounds for axis 0 with size 7

I think I will go with a manual copy over everything except the one param I want removed.

@eddiebergman
Copy link
Contributor Author

The copy approach for the record:

def remove_hyperparameter(name: str, space: ConfigurationSpace) -> ConfigurationSpace:
    """A new configuration space with the hyperparameter removed

    Essentially copies hp over and fails if there is conditionals or forbiddens
    """
    if name not in space._hyperparameters:
        raise ValueError(f"{name} not in {space}")

    # Copying conditionals only work on objects and not named entities
    # Seeing as we copy objects and don't use the originals, transfering these
    # to the new objects is a bit tedious, possible but not required at this time
    # ... same goes for forbiddens
    assert name not in space._conditionals, "Can't handle conditionals"
    assert not any(
        name != f.hyperparameter.name for f in space.get_forbiddens()
    ), "Can't handle forbiddens"

    hps = [copy(hp) for hp in space.get_hyperparameters() if hp.name != name]

    if isinstance(space.random, np.random.RandomState):
        new_seed = space.random.randint(2 ** 32 - 1)
    else:
        new_seed = copy(space.random)

    new_space = ConfigurationSpace(
        # TODO: not sure if this will have implications, assuming not
        seed=new_seed,
        name=copy(space.name),
        meta=copy(space.meta),
    )
    new_space.add_hyperparameters(hps)
    return new_space

eddiebergman added a commit to automl/mf-prior-bench that referenced this issue Aug 24, 2022
Please see this issue on ConfigSpace:
automl/ConfigSpace#267
@herilalaina herilalaina linked a pull request Jul 7, 2023 that will close this issue
@simonprovost
Copy link

+1 for this feature handling removing conditions and stuff like that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants