Normalization of input data to the unit cube

[jduerholt]

Hi all,

I have a question regarding a best practice:

The priors in botorch are built in a way that it is assumed that data is normalized to the unit cube (between zero and one). I usually realize this by using the Normalize input transform. The Normalize input transform can be configured to learn the bounds from the data during training, or one can provide it with the desired bounds.

Assume that one has an input x1 and historically it was varied between 5 and 10. If the bounds are learned, 5 would correspond to zero and 10 to 1 after applying the transform. This is definitely fine when one want to stay in the region between 5 and 10. But what would you recommend to do if one wants to vary it in the optimization between 3 and 15? Would you then setup the bounds for training the initial GP between between 3 and 15 or stay with the learned ones (5 and 10).

Currently, I am always taking the max(upper_bound, historical_data) as upper bound and min(lower_bound, historical_data) as lower bound. What is your recommended way of doing it?

Best,

Johannes

[hvarfner]

Hi @jduerholt!
So if I understand you correctly, the bounds will vary within a run and will, at their largest, be at 3 and 15. However, a substantial amount of the data will be acquired with the bounds set at 5 and 10, with you going back and forth between them?

As for what we currently do in Ax: I'm pretty sure the bounds would dynamically adjust to your current search space. This would potentially leave some data outside the unit cube, but it would still be included in the GP fitting if desired. We haven't specifically ablated the bounds' interaction with the lengthscale prior in this case, but I would be very surprised if a 2x change in one dimension would substantially impact performance.

With that said, I would keep the bounds at 5 and 10 throughout if you think this is the most relevant range for the aforementioned dimension. Otherwise, I would set them to 3 and 15.

What's the reason is for the changing bounds? Sounds interesting.

[jduerholt]

Usecase is kind of common in materials informatics. You have historical data obtained by some people often by some trial and error experimentation, now one wants to initialize a BO based on this original historical data, and for the BO campaling the bounds can be different to the bounds in the historical data and I am asking myself which lower and upper values one should use for the GP fitting. Currently, I always take the maximal possible bounds to prentent that any value lower than zero or larger than 1 enters the GP, but I never did any ablation ...

[jduerholt]

But the way, I am currently doing it can yield in some edge cases very complicated code, so I am thinking to just use the feature to learn the bounds from the data ..., any ideas?

[Balandat]

I agree with Carl here that I would be surprised if data outside the unit normalization had material impacts on the model fit (unless it's extremely far away, i.e. order-of-magnitude different). The main thing is to make sure to be consistent w.r.t. normalization and un-normalization -- you don't want to use one scaling for modeling and another one for scaling the search space bounds.

[jduerholt]

Thanks both for you replies, I was thinking in the same direction but was interested in your opinion ;)