About reduction_factor_schedule

[taylorlu]

Hi, thanks for sharing this great work.
I want to ask the training skill why we need to use dynamic input length in decode module, the relative variables self.max_r and self.r can be found in models.py.
The purpose seems to let it harder to train in the beginning since we only use less data to predict the whole mel sequence, but getting easier when the reduction_factor_schedule changes smaller which indicates larger input length. It looks a bit like simulated annealing algorithm, does it really work as I described? What will happen when self.max_r and self.r not the same.

[myagues]

Your intuition is right!
You will use large values for reduction_factor, at the start of the training, because missing data will make the model to rely on attention alignments. You can also imagine it as a type of dropout for auto-regressive models. Then, you can begin lowering the reduction_factor, which will improve the predicted mel spectrogram details, because the model will have more information. Here is an explanation using a Tacotron2 model.

What will happen when self.max_r and self.r not the same.

You need your model layers to be shape static, so you will initialize your projection output with the largest value of reduction_factor_schedule:

TransformerTTS/model/models.py

Line 83 in e4ded5b

self.final_proj_mel = tf.keras.layers.Dense(self.mel_channels * self.max_r, name='FinalProj')

When you reduce the value of self.r during your training, your layer will be the same size, but you will select just a part of it:

TransformerTTS/model/models.py

Lines 148 to 151 in e4ded5b

	out_proj = self.final_proj_mel(dec_output)[:, :, :self.r * self.mel_channels]
	b = int(tf.shape(out_proj)[0])
	t = int(tf.shape(out_proj)[1])
	mel = tf.reshape(out_proj, (b, t * self.r, self.mel_channels))

[taylorlu]

Thanks for your elaboration.

[cfrancesco]

Thank you @myagues, excellent explanation.