Confidence metric for targets

[nihaase]

Is it possible to provide a confidence value indicating how likely it is to find a better target value? For example, imagine you have 100 possible parameter combinations to identify the optimal target. Without Bayesian optimization, after conducting 20 experiments, one might estimate roughly a 20% confidence that the best parameter set has already been found. Intuitively, this confidence should be higher when using Bayesian optimization.

While the existing campaign stopping feature addresses stopping criteria, I would prefer a single confidence metric after each iteration. This would help users gauge whether, for instance, 80% or 95% confidence is sufficient to stop the search. Providing such a value would offer valuable guidance and improve decision-making during optimization.

[AdrianSosic]

Hi @nihaase, before we dive into details, have you seen the other example that tracks the probability of improvement? If so, what piece is missing for you to make it work?

[Scienfitz]

with PI its probably straightforward to compute

$$\text{confidencescore} = 1 - max_{x \in \text{candidates}} PI(x)$$

for other acqfs on could follow the same approach but would have to normalize by user-secified ranges like

$$\text{confidencescore} = 1 - \frac{max_{x \in \text{candidates}}EI(x)}{\text{desiredimprovementrange}}$$

$$\text{confidencescore} = 1 - \frac{max_{x \in \text{candidates}}UCB(x) - f_{\text{best}}}{\text{desiredimprovementrange}}$$

Instead of an improvement range one could probably also phrase this in terms of an absolute desired best value.

whether any of this would work well is not clear and needs to be tested, also this is only possible for single-targets. For multi target you should prob estimate one confidence score per target

[Scienfitz]

is this a case for a ConfidenceInsight or ImprovementInsight?

[AdrianSosic]

is this a case for a ConfidenceInsight or ImprovementInsight?

Since many people seem to want an easy interface for it – probably yes 😬

[AdrianSosic]

Let's put it on our TODO list. @nihaase: could you perhaps share some thoughts about the ideal interface you'd wish for? E.g. in terms of some python signatures?

[nihaase]

I think the PI is already a useful tool for this purpose; however, I would prefer a metric that is easier to understand and more accessible. Since many users may be non-technical, having a straightforward metric would be beneficial. You might even consider introducing different categories, such as: How likely is it to find better parameters? — very high, high, medium, low, very low.

Do yo have any experience how interpretable the PI currently is and if it is possible to generalize the interpretation over different use cases?

For multiple targets, it may be possible to calculate a single metric by taking a weighted sum of the targets and then averaging the result. This would provide one consolidated value that reflects the overall performance across all targets.

$$\text{Metric} = \frac{1}{n} \sum_{i=1}^{n} w_i \cdot T_i $$