fp8 and bfloat16 support

[lostmsu]

NVidia actually has two variants of fp8 with different sizes of mantissa/exponent. bfloat16 is also unique. There's also TensorFloat32 which is really more like bfloat19. Perhaps it would make sense to have float<SizeOfMantissa, SizeOfExponent> generic type (hackery).

[MoFtZ]

It looks like Cuda provides a few alternate floating point options, including bf16 and tf32.

This would have to be a Cuda only feature, as there is no equivalent in OpenCL 2.0.

We already have support for Half. Would we add support for BFloat16 and TensorFloat32 types? @m4rs-mt

[lostmsu]

@MoFtZ that's why it might make sense to have the generic-sized float type as I mentioned, with boolean guards. E.g. bool Accelerator.SupportsType(...) function, which would let user choose a different kernel.

[m4rs-mt]

@lostmsu @MoFtZ I think this makes sense to add given that NVIDIA GPUs can take serious advantage of these types. I wonder how we can make this happen in a convenient way. Let's get into more detail on Thursday in our talk-to-dev session.

[MoFtZ]

Based on our last discussions, this is more broadly related to adding support for the Cuda WMMA (Warp Level Matrix Multiply-Accumulate Instructions); adding support for the fp8 and bfloat16 types is not very useful without WMMA support.