Classifier-free guidance resolution weighting

[snatchysquid]

In section 3.4, the ControlNet paper talks about CFG-RW, quoting:

In challenging cases, e.g., when no prompts are given, adding it to both ϵuc and ϵc will completely remove CFG guidance (Figure 5b);
using only ϵc will make the guidance very strong (Figure 5c).
Our solution is to first add the conditioning image to ϵ_c and then multiply a weight wi to each connection between Stable Diffusion and ControlNet according to the resolution of each block wi = 64/hi, where hi is the size of i th block, e.g., h1 = 8, h2 = 16, ..., h13 = 64

I don't quite understand what this means and where is this implemented in the code in this repository. My questions, therefore, are as follows:

1. Do I understand correctly that what we do is train the model without any weighting, and then for the ϵ_uc we use unconditional SD without ControlNet, and for ϵ_c we use ControlNet but before adding the skip connection, we multiply the output by wi (meaning SD_Layer_i_final_output = SD_Layer_i_output + (w_i*ControlNet_Layer_i_output) )?
2. If so, what is the logic and motivation for doing that? It doesn't sound trivial that this would be want we want to do.
3. Finally, I'd like to know where is this implemented in the code?

TAGS: @lllyasviel @scarbain @williamyang1991

[geroldmeisinger]

you again... shouldn't you be preparing images? ;)

I don't know much about the theory but I think they are referring to this: #188
and your question may also be related to the zero-convolutions stuff, look here #550 (comment)
If you want a definitive answer, you should @ the original authors.

[snatchysquid]

Hey gerold! I already downloaded the environment and trained an example model of my own, all that's left is prepare the dataset.
BTW oct31 is also christmas in decimal. Good to see you :) Thanks for the references, I'll check them out
EDIT: Seems like these links don't really answer my question, not really related to zero-convs (with that topic I feel comfortable)

[Chen-Suyi]

I have the same question as yours and found the implementation of CFG-RW in the codes.
It's here.

However, It seems the weights have been fixed to 1, which differs from CFG-RW discussed in the paper.

[zhangchi2004]

Thanks! I am just looking for it. In my mind it seems the weights should be fixed to 1 during training (so unweighted control are used) and set to other values during sampling, so reweighted cfg is achieved.