pipeline fail to move to "cuda" if one of the component is PeftModel

[Erisura]

When I was doing infer, I loaded the pre-trained weights using the following method:

self.transformer = PeftModel.from_pretrained(
            self.base_transformer,
            lora_model_path
        )

and loaded the pipeline in the following way:

pipe = FluxControlNetPipeline(transformer=transformer, ......)

However, I encountered an error when running it:
ValueError: It seems like you have activated sequential model offloading by calling enable_sequential_cpu_offload, but are now attempting to move the pipeline to GPU. This is not compatible with offloading. Please, move your pipeline .to('cpu') or consider removing the move altogether if you use sequential offloading.

Strangely, I have never manually set sequential model offloading. And I found that if I didn't call

self.transformer =  PeftModel.from_pretrained(
            self.base_transformer,
            lora_model_path
        )

Instead, directly set self.transformer = self.base_transformer, and the subsequent code remains unchanged, the above error will not occur.
If there's someone who has encountered the same issue?

System Info

diffusers==0.33.0.dev0
peft==0.14.0

@yiyixuxu @sayakpaul @DN6

[sayakpaul]

Can you provide a minimal reproducible snippet? The code is not reproducible.