🎉 Flash-DMA v0.1.0 - Initial Release

[LoserCheems]

We're excited to announce the first official release of Flash-DMA (Flash Dynamic Mask Attention)!

🚀 What is Flash-DMA?

Flash-DMA is a high-performance attention implementation that combines:

• Flash Attention's memory efficiency
• Dynamic Mask Attention's sparse computation
• Support for extremely long sequences (128K+ tokens)

✨ Key Features

🔥 Performance

• Sparse Attention: Reduces computation from O(N²) to O(N·w) where w ≪ N
• Memory Efficient: Maintains O(N) memory complexity
• CUDA Accelerated: Custom sparse GEMM operations at kernel level

🛠️ Multiple Backends

• CUDA Backend: Maximum performance with custom kernels
• Triton Backend: Flexibility for research and development
• Flex Backend: Integration with Transformers library

📏 Long Sequence Support

• Efficiently handles sequences of 128K+ tokens
• Dynamic masking when sequence length exceeds keep_window_size
• Optimized memory layouts for large-scale processing

📦 Installation

Prerequisites

• Python 3.9+
• PyTorch 2.0+
• CUDA 11.8+
• NVIDIA GPU with Compute Capability 8.0+

Install from Source

git clone https://github.com/SmallDoges/flash-dmattn.git
cd flash-dmattn
git submodule update --init --recursive
pip install .

<hr /><em>This discussion was created from the release <a href='https://github.com/SmallDoges/flash-dmattn/releases/tag/'>🎉 Flash-DMA v0.1.0 - Initial Release</a>.</em>