Problem with SingleTaskGP

[ToennisStef]

I encountered a Problem while working with the SingleTaskGP model.
I wanted to retrun the multivariate distribution by simply computing model(x)
but i got an error message regarding a dtype mismatch. The whole error message is:

{
	"name": "RuntimeError",
	"message": "expected m1 and m2 to have the same dtype, but got: double != float",
	"stack": "---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[19], line 2
      1 model.eval()
----> 2 model(x)

File c:\\ProgramData\\Anaconda3\\envs\\BO_init\\Lib\\site-packages\\gpytorch\\models\\exact_gp.py:333, in ExactGP.__call__(self, *args, **kwargs)
    328 # Make the prediction
    329 with settings.cg_tolerance(settings.eval_cg_tolerance.value()):
    330     (
    331         predictive_mean,
    332         predictive_covar,
--> 333     ) = self.prediction_strategy.exact_prediction(full_mean, full_covar)
    335 # Reshape predictive mean to match the appropriate event shape
    336 predictive_mean = predictive_mean.view(*batch_shape, *test_shape).contiguous()

File c:\\ProgramData\\Anaconda3\\envs\\BO_init\\Lib\\site-packages\\gpytorch\\models\\exact_prediction_strategies.py:289, in DefaultPredictionStrategy.exact_prediction(self, joint_mean, joint_covar)
    285     test_test_covar = joint_covar[..., self.num_train :, self.num_train :]
    286     test_train_covar = joint_covar[..., self.num_train :, : self.num_train]
    288 return (
--> 289     self.exact_predictive_mean(test_mean, test_train_covar),
    290     self.exact_predictive_covar(test_test_covar, test_train_covar),
    291 )

File c:\\ProgramData\\Anaconda3\\envs\\BO_init\\Lib\\site-packages\\gpytorch\\models\\exact_prediction_strategies.py:309, in DefaultPredictionStrategy.exact_predictive_mean(self, test_mean, test_train_covar)
    307     res = (test_train_covar @ self.mean_cache.squeeze(1).unsqueeze(-1)).squeeze(-1)
    308 else:
--> 309     res = (test_train_covar @ self.mean_cache.unsqueeze(-1)).squeeze(-1)
    310 res = res + test_mean
    312 return res

File c:\\ProgramData\\Anaconda3\\envs\\BO_init\\Lib\\site-packages\\linear_operator\\operators\\_linear_operator.py:2897, in LinearOperator.__matmul__(self, other)
   2891 def __matmul__(
   2892     self: Float[LinearOperator, \"*batch M N\"],
   2893     other: Union[
   2894         Float[torch.Tensor, \"*batch2 N D\"], Float[torch.Tensor, \"N\"], Float[LinearOperator, \"*batch2 N D\"]
   2895     ],
   2896 ) -> Union[Float[torch.Tensor, \"... M D\"], Float[torch.Tensor, \"... M\"], Float[LinearOperator, \"... M D\"]]:
-> 2897     return self.matmul(other)

File c:\\ProgramData\\Anaconda3\\envs\\BO_init\\Lib\\site-packages\\linear_operator\\operators\\_linear_operator.py:1831, in LinearOperator.matmul(self, other)
   1827     from .matmul_linear_operator import MatmulLinearOperator
   1829     return MatmulLinearOperator(self, other)
-> 1831 return Matmul.apply(self.representation_tree(), other, *self.representation())

File c:\\ProgramData\\Anaconda3\\envs\\BO_init\\Lib\\site-packages\\torch\\autograd\\function.py:539, in Function.apply(cls, *args, **kwargs)
    536 if not torch._C._are_functorch_transforms_active():
    537     # See NOTE: [functorch vjp and autograd interaction]
    538     args = _functorch.utils.unwrap_dead_wrappers(args)
--> 539     return super().apply(*args, **kwargs)  # type: ignore[misc]
    541 if cls.setup_context == _SingleLevelFunction.setup_context:
    542     raise RuntimeError(
    543         \"In order to use an autograd.Function with functorch transforms \"
    544         \"(vmap, grad, jvp, jacrev, ...), it must override the setup_context \"
    545         \"staticmethod. For more details, please see \"
    546         \"https://pytorch.org/docs/master/notes/extending.func.html\"
    547     )

File c:\\ProgramData\\Anaconda3\\envs\\BO_init\\Lib\\site-packages\\linear_operator\\functions\\_matmul.py:21, in Matmul.forward(ctx, representation_tree, rhs, *matrix_args)
     18     is_vector = False
     20 linear_op = ctx.representation_tree(*matrix_args)
---> 21 res = linear_op._matmul(rhs)
     23 to_save = [orig_rhs] + list(matrix_args)
     24 ctx.save_for_backward(*to_save)

File c:\\ProgramData\\Anaconda3\\envs\\BO_init\\Lib\\site-packages\\linear_operator\\operators\\dense_linear_operator.py:65, in DenseLinearOperator._matmul(self, rhs)
     61 def _matmul(
     62     self: Float[LinearOperator, \"*batch M N\"],
     63     rhs: Union[Float[torch.Tensor, \"*batch2 N C\"], Float[torch.Tensor, \"*batch2 N\"]],
     64 ) -> Union[Float[torch.Tensor, \"... M C\"], Float[torch.Tensor, \"... M\"]]:
---> 65     return torch.matmul(self.tensor, rhs)

RuntimeError: expected m1 and m2 to have the same dtype, but got: double != float"
}

Previously i worked with a custom Botorch GP model and created a class which inherents from ´gpytorch.models.ExactGP´. I also set the default data type to double with ´torch.set_default_dtype(torch.double)´

The code to reproduce the error is:

import gpytorch
import torch
import botorch

def func(x):
    return 1 + torch.sin(x[..., :1] )

res = 10000
lowerBound = -1
upperBound = 1
x = torch.linspace(lowerBound,upperBound,res,dtype=torch.double)
y_true = func(x.reshape([-1,1]))

from torch.quasirandom import SobolEngine
from botorch.models import SingleTaskGP
from botorch.acquisition import qExpectedImprovement

N_INIT = 1
DIM = 1
X = SobolEngine(dimension=DIM, scramble=True, seed=0).draw(N_INIT)
Y = func(X)
model = SingleTaskGP(X,Y,likelihood=likelihood)

model(x)

The custom GP class that i used was:

class ExactGP(gpytorch.models.ExactGP, botorch.models.gpytorch.GPyTorchModel):

    _num_outputs = 1  # to inform GPyTorchModel API

    def __init__(self, train_X, train_Y, Likelihood): 
        super().__init__(train_X, train_Y, Likelihood)
        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

Did somebody encounter this problem before? Please let me know if you have any tips on how to solve this problem. :)

[ToennisStef]

nevermind i think i could solve the problem.
Even when i set torch.set_default_dtype(torch.double) the sobol samples returned a float32 typ so i had to define the dtype with:
X = SobolEngine(dimension=DIM, scramble=True, seed=0).draw(N_INIT,dtype=torch.double)
This seems to have fixed the problem.

I still don't know why the ExactGP class could handle the different datatypes though...

[saitcakmak]

Hi @ToennisStef!

Even when i set torch.set_default_dtype(torch.double) the sobol samples returned a float32 typ

I tried to reproduce this but got samples of the correct dtype.

Though I found some Sobol utilities in BoTorch that use float32 by default while looking into this. I'll put up a fix for those.

[saitcakmak]

Looks like there are issues with pytorch documentation / behavior here. According to the source code & documentation, it should be returning float32 by default but it doesn't: https://pytorch.org/docs/stable/generated/torch.quasirandom.SobolEngine.html#torch.quasirandom.SobolEngine.draw

Following up on this here: pytorch/pytorch#126478