Parameter Setup for Transformer-Big

[ymjiang]

Question in short: In train_transformer.py, currently the default setting is for Transformer-Base. If I want to train Transformer-Big, what parameter should I change?

I refer to the Transformer's paper and find out the major setup differences between Transformer-BIG and Transformer-BASE are num of heads, num of dimensions and filter dimensions. Then I try to find the associated parameters in train_transformer.py script, and confirm --num_heads is for num of heads, while num_units is for num of dimensions. But I cannot determine what parameter I can change to set filter dimensions.

Tensorflow seems to have the option to change filter dimension: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py#L1837

[szhengac]

In our script, it is --hidden_size. You may also need to change dropout and epsilon accordingly.

[eric-haibin-lin]

@ymjiang I am interested in your transformer-big result. Feel free to share if you are able to reproduce it, or hit any blocker. BTW I'm adding fp16 support to the transformer encoder block in #505 which may benefit your work, too

[ymjiang]

Hi @eric-haibin-lin , unfortunately I haven't succeed in training Transformer-big to reasonable BLEU so far (my current setup only converges to BLEU=3.2 on WMT2016BPE en-de).

Default Transformer-base param: (this can converge to BLEU=25)
--adam_beta2 0.98 --hidden_size 2048 --num_heads 8 --num_units 512 --dropout 0.1

My Transformer-big param: (only converge to BLEU=3.2)
--adam_beta2 0.998 --hidden_size 4096 --num_heads 16 --num_units 1024 --dropout 0.3

I tried to tune the learning rate (by setting it to reasonably smaller) but still get similar result, so I suspect it might be due to something else wrong rather than the learning rate.

[szha]

@ymjiang how does the log look? We might be able to offer some help.

[ymjiang]

@szha Sorry I am currently engaged with another task. Will get back here once I am free.