Bugged recommendation cache check accessing `allow_recommending_already_recommended` when it's `UNSPECIFIED`

[juliaphilipp]

Hi BayBe Team,

I am running in (for me) unexpected behavior when getting recommendations from a campaign with a continuous search space. The issue happens always when getting the second round of recommendations, independently of whether I add measurements before or not.

The error I get is a "NotImplementedError: 'UNSPECIFIED' has no Boolean representation." Pointing towards allow_recommending_already_recommended. When I try to set allow_recommending_already_recommended to False in the campaign, I get the following ValueError as expected: "ValueError: For search spaces of type other than 'SearchSpaceType.DISCRETE', 'allow_recommending_already_recommended' cannot be set since the flag is meaningless in such contexts."

Below a minimal example that produces the described error, using baybe 0.14.2. I'm sure I'm just missing something, I'd appreciate any input :)

from baybe.targets import NumericalTarget

from baybe.objectives import (
    ParetoObjective
)

from baybe.parameters import (
    NumericalContinuousParameter,
)
from baybe.constraints import (
    ContinuousLinearConstraint
)

from baybe.searchspace import SearchSpace

from baybe import Campaign
from baybe.utils.random import temporary_seed

from baybe.utils.dataframe import add_fake_measurements, add_parameter_noise

t1 = NumericalTarget(name="TargetA", minimize=False)
t2 = NumericalTarget(name="TargetB", minimize=True)

objective = ParetoObjective(targets=[t1, t2])

parameters = [
    NumericalContinuousParameter(
        name="ComponentA",
        bounds=(20,70)
    ),
    NumericalContinuousParameter(
        name="ComponentB",
        bounds=(0,100)
    ),
    NumericalContinuousParameter(
        name="ComponentC",
        bounds=(0, 100)
    ),
    NumericalContinuousParameter(
        name="ComponentD",
        bounds=(1,5)
    ),
    
]

constraints = [
    ContinuousLinearConstraint(
        parameters=["ComponentA","ComponentB", "ComponentC","ComponentD"], 
        operator="=", 
        coefficients=(1.0, 1.0, 1.0,1.0), 
        rhs=100
    ),
]

searchspace = SearchSpace.from_product(parameters=parameters, constraints=constraints)

campaign = Campaign(
    searchspace=searchspace,
)

with temporary_seed(123):
    recommendation = campaign.recommend(batch_size=12)
    
measurements = recommendation.copy()
add_fake_measurements(measurements, campaign.targets)
add_parameter_noise(measurements, campaign.parameters)

new_recommendation = campaign.recommend(batch_size=12)

[Scienfitz]

Hi @juliaphilipp

I can reproduce the error being thrown on 0.14.2 and main, hence I labeled this issue as potential bug.

I notice one strange thing: You are not assigning the created objective to the campaign. This is something that we support because there are recommenders that do not operate on measurements and just ignore them, hence they don't have a need to specify targets. But overall, this will not be the main path most suers would take. Can you comment why no objective was passed? I would expect this

campaign = Campaign(
    searchspace=searchspace,
    objective=objective,
)

If I pass the objective + use campaign.add_measurements(measurements) I get no error and the second recommendation runs as expected (takes also longer time because the second request after measurement addition is model based unlike the first request).

If you don't add the objective but add measurements you get NotImplementedError: Recommenders of type 'BayesianRecommender' require that an objective is specified. which is expected.

Whenever I do not add measurements and ask for the second round I get the error you describe. I think there is a bugged interaction with caching, which will be investigated further. The recommendations returned should simply be the one from the first round independent of any calculation etc.

If you want to use your code for production I think adding the objective should already unblock you.

[juliaphilipp]

Hi @Scienfitz ,
thank you for the quick response!

The lack of passing the objective to the campaign was poor copy-paste on my end, apologies! I am actually experiencing the error as described above even when passing the objective to the campaign like you described.

Would you like me to update the example above?

[Scienfitz]

no need to update, thats consistent:

• I get your error whenever I do not add the measurements (independent of the objective thing)
• When measurements are actually added the results are as expected (both for adding the objective or not)

I think we have a bug in this line in campaign.py

        if (
            pending_experiments is None
            and (cache := self._cached_recommendation) is not None
            and self.allow_recommending_already_recommended # <- problem
            and len(cache) == batch_size
        ):
            return cache

self.allow_recommending_already_recommended is not a simple boolean but a more complex object that can be unspecified in which case it cannot be evaluated in such an expression. A simple solution would be to expand the if condition to check the searchspace type but im pinging @AdrianSosic because he implemented the unspecified object and maybe sees a more elegant solution.

[juliaphilipp]

Thank you for your help :)

[AdrianSosic]

Hey @juliaphilipp and @Scienfitz. Yes, can confirm this is a bug, I've noticed this already a while ago but haven't prioritized the fix since it does not arise in the normal workflow, as @Scienfitz correctly pointed out (i.e. when measurements are added) but only in a this particular cache hit situation.

Since the suggestion should unblock you now, I suggest we fix it as part of the new 0.15.0 release. My current idea would be (and that's the other reason why I waited) to not fix the existing logic, but replace the UNSPECIFIED object with the more flexible AutoBool that the settings PR ships, which should be perfectly suitable for these flags.

[AdrianSosic]

Hi @juliaphilipp. I've just published a version 0.14.3 that ships a hotfix for the problem. Can you perhaps briefly confirm that everything works for you as expected? A proper fix is already scheduled for 0.15.0 (#742)

[juliaphilipp]

Hi @AdrianSosic - sorry for the delay, I just tried it out and it works great! Thank you :)

[AdrianSosic]

Keeping it open until the proper 0.15.0 fix is merged ;)