Assistance on implementing Flash Attention 2 for Turing

[samuelzxu]

Hi All,

I've seen that there are commented lines flash-attention/csrc/flash_attn/flash_api.cpp in the codebase suggesting there have been attempts in the past at getting flash attention 2 working. This also suggests that there have been some significant barriers to this effort.

I'd like to try my hand at this task, but would really appreciate insights the authors or any other readers might have on this topic. What are the most significant obstacles? Is it the architecture-specific optimizations, dev time, tiling, or something else?