📚Tensor/CUDA Cores, 📖150+ CUDA Kernels, 🔥🔥toy-hgemm library with WMMA, MMA and CuTe(99%~100%+ TFLOPS of cuBLAS 🎉🎉).
-
Updated
Nov 27, 2024 - Cuda
📚Tensor/CUDA Cores, 📖150+ CUDA Kernels, 🔥🔥toy-hgemm library with WMMA, MMA and CuTe(99%~100%+ TFLOPS of cuBLAS 🎉🎉).
Efficient kernel for RMS normalization with fused operations, includes both forward and backward passes, compatibility with PyTorch.
Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.
Simple character level Transformer
Generative models nano version for fun. No STOA here, nano first.
Add a description, image, and links to the rmsnorm topic page so that developers can more easily learn about it.
To associate your repository with the rmsnorm topic, visit your repo's landing page and select "manage topics."