Why use start letter(0) replace the first position of data?

[ElroyLR]

hello suragnair,

I have run your code and observed that during pretraining the generator, in the 'helper.py', function 'prepare_generator_batch', the start data of input is replaced to zero as a start letter. I am curious about the purpose of this operation and how it influences the loss calculation.

Looking forward to your reply. Thank you.