Gemma-2 architecture support

[arbi-dev]

Hi, just curious how complicated supporting Gemma-2-27b and 9b architectures with exl2 is likely to be? I'm sure @turboderp is already on it :)

[Vhallo]

Don't think there's much rush? Looks like Flash Attention will take at least some time to properly support it too.

[turboderp]

I am indeed already on it. There's been a lot of confusion around Gemma2, even more than the first Gemma, and it's not even clear at this moment what a correct implementation looks like. The HF Transformers support is incomplete at best, perhaps even straight up broken for longer than 4k context length. It's somewhere near the top of my list due to popular demand, but it's going to take some time to even decide on a correct reference implementation to adhere to.

Also as pointed out, flash-attn will need to implement soft-capping. This will probably happen soon, but I obviously can't say for sure.

[arbi-dev]

Thanks for the reply! Some new info on softcapping etc here: https://unsloth.ai/blog/gemma2

[turboderp]

The softcapping is very straightforward, at least as far as inference goes.

The Gemma2 implementation is finished, too. The only thing missing for full support is this PR in flash-attn. I'm hesitant to push the changes until then, since models aren't going to quantize correctly without it.