Small fixes and refactoring

[mseeger]

Mostly refactoring and small fixes:

• config.sliding_window_layer_period instead of overwriting config.sliding_window_layer_placing (str to int)
• Comment on config.attention_logit_softcapping: Will hit performance if this is used
• cos, sin passed down in GPT.forward must have batch dimension if input_pos is None. This probably worked because PyTorch is adding singleton dimensions at the front of a shape for broadcasting, but it did confuse me.
• Block.forward: Missed self.post_mlp_norm if parallel_residual. BTW: None of your models have parallel_residual = True, is anybody using this?

Not doing anything new. Feel free to reject, but this would help to make the code easier to understand.

[Andrei-Aksionov]

Hello @mseeger

Thanks for the PR!

Block.forward: Missed self.post_mlp_norm if parallel_residual.

Nice catch! 🙂

BTW: None of your models have parallel_residual = True, is anybody using this?

Some older models supported parallel_residual. In #1821 some of such models were dropped. In one of the future PRs, I guess, we will drop an unused code. That, for instance, should make MLP class easier to read.

[mseeger]

How does it work for this project? I suppose somebody needs to review. I don't see why the one test (Thunder) is failing.

[Andrei-Aksionov]

Hello @mseeger

Thanks for the PR. Looks good 🚀.
I only left a couple of nits.

I don't see why the one test (Thunder) is failing.

I'll fix it today.

[mseeger]

OK, incorporated requested changes

[Andrei-Aksionov]

Hello @mseeger

Sorry for the delay.
Now the code is ready to merge.

Thanks again for the PR! 🚀