Build your own ML framework. Start small. Go deep.
📚 Read the Interactive Course →
Most ML education teaches you to use frameworks. TinyTorch teaches you to build them.
Traditional ML Course: TinyTorch Approach:
├── import torch ├── class Tensor:
├── model = nn.Linear(10, 1) │ def __add__(self, other): ...
├── loss = nn.MSELoss() │ def backward(self): ...
└── optimizer.step() ├── class Linear:
│ def forward(self, x):
│ return x @ self.weight + self.bias
├── def mse_loss(pred, target):
│ return ((pred - target) ** 2).mean()
├── class SGD:
│ def step(self):
└── param.data -= lr * param.grad
Go from "How does this work?" 🤷 to "I implemented every line!" 💪Result: You become the person others come to when they need to understand "how PyTorch actually works under the hood."
- No black boxes: Implement every component from scratch
- Immediate ownership: Use YOUR code in real neural networks
- Deep understanding: Know exactly how each piece works
- Professional workflow: Development with
titoCLI, automated testing - Real datasets: Train on CIFAR-10, not toy data
- Production patterns: MLOps, monitoring, optimization from day one
- Start simple: Implement
hello_world()function - Build systematically: Each module enables the next
- End powerful: Deploy production ML systems with monitoring
- Code works immediately: No waiting to see results
- Visual progress: Success indicators and system integration
- "Aha moments": Watch your
ReLUpower real neural networks
- One Complete ML Framework — Not 14 separate exercises, but integrated components building into your own PyTorch-style toolkit
- Fully Functional System — Every piece connects: your tensors power your layers, your autograd enables your optimizers, your framework trains real networks
- Real Applications — Train neural networks on CIFAR-10 using 100% your own code, no PyTorch imports
- Production-Ready Skills — Complete ML lifecycle: data loading, training, optimization, deployment, monitoring
- Deep Systems Understanding — Know exactly how every component works and integrates because you built it all
git clone https://github.com/mlsysbook/TinyTorch.git
cd TinyTorch
pip install -r requirements.txt # Install all dependencies (numpy, jupyter, pytest, etc.)
pip install -e . # Install TinyTorch package in editable mode
tito system doctor # Verify your setup
cd modules/source/01_setup
jupyter lab setup_dev.py # Launch your first module# System check
tito system info
tito system doctor
# Module workflow
tito export 01_setup
tito test 01_setup
tito nbdev build # Update packageTinyTorch/
├── modules/source/ # 16 educational modules
│ ├── 01_setup/ # Development environment setup
│ │ ├── module.yaml # Module metadata
│ │ ├── README.md # Learning objectives and guide
│ │ └── setup_dev.py # Implementation file
│ ├── 02_tensor/ # N-dimensional arrays
│ │ ├── module.yaml
│ │ ├── README.md
│ │ └── tensor_dev.py
│ ├── 03_activations/ # Neural network activation functions
│ ├── 04_layers/ # Dense layers and transformations
│ ├── 05_dense/ # Sequential networks and MLPs
│ ├── 06_spatial/ # Convolutional neural networks
│ ├── 07_attention/ # Self-attention and transformer components
│ ├── 08_dataloader/ # Data loading and preprocessing
│ ├── 09_autograd/ # Automatic differentiation
│ ├── 10_optimizers/ # SGD, Adam, learning rate scheduling
│ ├── 11_training/ # Training loops and validation
│ ├── 12_compression/ # Model optimization and compression
│ ├── 13_kernels/ # High-performance operations
│ ├── 14_benchmarking/ # Performance analysis and profiling
│ ├── 15_mlops/ # Production monitoring and deployment
│ └── 16_capstone/ # Systems engineering capstone project
├── tinytorch/ # Your built framework package
│ ├── core/ # Core implementations (exported from modules)
│ │ ├── tensor.py # Generated from 02_tensor
│ │ ├── activations.py # Generated from 03_activations
│ │ ├── layers.py # Generated from 04_layers
│ │ ├── dense.py # Generated from 05_dense
│ │ ├── spatial.py # Generated from 06_spatial
│ │ ├── attention.py # Generated from 07_attention
│ │ └── ... # All your implementations
│ └── utils/ # Shared utilities and tools
├── book/ # Interactive course website
│ ├── _config.yml # Jupyter Book configuration
│ ├── intro.md # Course introduction
│ └── chapters/ # Generated from module READMEs
├── tito/ # CLI tool for development workflow
│ ├── commands/ # Student and instructor commands
│ └── tools/ # Testing and build automation
└── tests/ # Integration tests
How It Works:
- Develop in
modules/source/- Each module has a*_dev.pyfile where you implement components - Export to
tinytorch/- Usetito exportto build your implementations into a real Python package - Use your framework - Import and use your own code:
from tinytorch.core.tensor import Tensor - Test everything - Run
tito testto verify your implementations work correctly - Build iteratively - Each module builds on previous ones, creating a complete ML framework
Difficulty Progression: ⭐ Beginner → ⭐⭐ Intermediate → ⭐⭐⭐ Advanced → ⭐⭐⭐⭐ Expert → ⭐⭐⭐⭐⭐🥷 Capstone
- 01_setup: Development environment and CLI tools
- 02_tensor: N-dimensional arrays and tensor operations
- 03_activations: ReLU, Sigmoid, Tanh, Softmax functions
- 04_layers: Dense layers and matrix operations
- 05_dense: Sequential networks and MLPs
- 06_spatial: Convolutional neural networks and image processing
- 07_attention: Self-attention and transformer components
- 08_dataloader: Data loading, batching, and preprocessing
- 09_autograd: Automatic differentiation and backpropagation
- 10_optimizers: SGD, Adam, and learning rate scheduling
- 11_training: Training loops, metrics, and validation
- 12_compression: Model pruning, quantization, and distillation
- 13_kernels: Performance optimization and custom operations
- 14_benchmarking: Profiling, testing, and performance analysis
- 15_mlops: Monitoring, deployment, and production systems
- 16_capstone: Advanced framework engineering specialization tracks
Status: All 16 modules complete with inline tests and educational content
This isn't 16 isolated assignments. Every component you build integrates into one cohesive, fully functional ML framework:
flowchart TD
A[01_setup<br/>Setup & Environment] --> B[02_tensor<br/>Core Tensor Operations]
B --> C[03_activations<br/>ReLU, Sigmoid, Tanh]
C --> D[04_layers<br/>Dense Layers]
D --> E[05_dense<br/>Sequential Networks]
E --> F[06_spatial<br/>Convolutional Networks]
E --> G[07_attention<br/>Self-Attention]
F --> H[08_dataloader<br/>Data Loading]
B --> I[09_autograd<br/>Automatic Differentiation]
I --> J[10_optimizers<br/>SGD & Adam]
H --> K[11_training<br/>Training Loops]
G --> K
J --> K
K --> L[12_compression<br/>Model Optimization]
K --> M[13_kernels<br/>High-Performance Ops]
K --> N[14_benchmarking<br/>Performance Analysis]
K --> O[15_mlops<br/>Production Monitoring]
L --> P[16_capstone<br/>Framework Engineering]
M --> P
N --> P
O --> P
Foundation (01-05): Build your core data structures and basic operations
Deep Learning (06-10): Add neural networks and automatic differentiation
Production (11-15): Scale to real applications with training and production systems
Mastery (16): Optimize and extend your complete framework
The Result: A complete, working ML framework built entirely by you, capable of:
- ✅ Training CNNs on CIFAR-10 with 90%+ accuracy
- ✅ Implementing modern optimizers (Adam, learning rate scheduling)
- ✅ Deploying compressed models with 75% size reduction
- ✅ Production monitoring with comprehensive metrics
After completing the 15 core modules, you have a complete ML framework. The final challenge: make it better through systems engineering.
Choose Your Focus:
- ⚡ Performance Engineering: GPU kernels, vectorization, memory-efficient operations
- 🧠 Algorithm Extensions: Transformer layers, BatchNorm, Dropout, advanced optimizers
- 🔧 Systems Optimization: Multi-GPU training, distributed computing, memory profiling
- 📊 Benchmarking Analysis: Compare your framework to PyTorch, identify bottlenecks
- 🛠️ Developer Tools: Better debugging, visualization, error messages, testing
The Constraint: No import torch allowed. Build on your TinyTorch implementation. This demonstrates true mastery of ML systems engineering and optimization.
🔧 Build: Implement ReLU from scratch
def relu(x):
# YOU implement this function
return ??? # What should this be?🚀 Use: Immediately use your own code
from tinytorch.core.activations import ReLU # YOUR implementation!
layer = ReLU()
output = layer.forward(input_tensor) # Your code working!💡 Reflect: See it working in real networks
# Your ReLU is now part of a real neural network
model = Sequential([
Dense(784, 128),
ReLU(), # <-- Your implementation
Dense(128, 10)
])This pattern repeats for every component — you build it, use it immediately, then see how it fits into larger systems.
- Build every component from scratch
- Understand performance trade-offs
- See how engineering decisions impact ML outcomes
- Use real datasets (CIFAR-10, MNIST)
- Implement proper testing and benchmarking
- Learn MLOps and system design principles
- Each module builds on previous work
- Immediate feedback through inline testing
- Progressive complexity with solid foundations
- Live Site: https://mlsysbook.github.io/TinyTorch/
- Auto-updated from source code on every release
- Complete course content with executable examples
- Real implementation details with solution code
devbranch: Active development and experimentsmainbranch: Stable releases that trigger documentation deployment- Inline testing: Tests embedded directly in source modules
- Continuous integration: Automatic building and deployment
# Work on dev branch
git checkout dev
# Edit source modules
cd modules/source/02_tensor
jupyter lab tensor_dev.py
# Export to package
tito export 02_tensor
# Test your implementation
tito test 02_tensor
# Build complete package
tito nbdev build# Ready for release
git checkout main
git merge dev
git push origin main # Triggers documentation deploymentTinyTorch/
├── modules/source/XX/ # 16 source modules with inline tests
├── tinytorch/core/ # Your exported ML framework
├── tito/ # CLI and course management tools
├── book/ # Jupyter Book source and config
├── tests/ # Integration tests
└── docs/ # Development guides and workflows
- Python 3.8+ — Modern Python with type hints
- NumPy — Numerical foundations
- Jupyter Lab — Interactive development
- Rich — Beautiful CLI output
- NBDev — Literate programming and packaging
- Jupyter Book — Interactive documentation
- GitHub Actions — Continuous integration and deployment
Students who complete TinyTorch can:
✅ Build complete neural networks from tensors to training loops
✅ Implement modern ML algorithms (Adam, dropout, batch norm)
✅ Optimize performance with profiling and custom kernels
✅ Deploy production systems with monitoring and MLOps
✅ Debug and test ML systems with proper engineering practices
✅ Understand trade-offs between accuracy, speed, and resources
👉 Start Learning Now — Complete course in your browser
git clone https://github.com/mlsysbook/TinyTorch.git
cd TinyTorch
pip install -r requirements.txt # Install all dependencies (numpy, jupyter, pytest, etc.)
pip install -e . # Install TinyTorch package in editable mode
tito system doctor
cd modules/source/01_setup
jupyter lab setup_dev.py# Clone and verify system
git clone https://github.com/mlsysbook/TinyTorch.git
cd TinyTorch
tito system info
# Test module workflow
tito export 01_setup && tito test 01_setup🔥 Ready to build your ML framework? Start with TinyTorch and understand every layer. Start Small. Go Deep.
🚀 "Why not just use PyTorch/TensorFlow? This seems like reinventing the wheel."
You're right - for production, use PyTorch! But consider:
🤔 Deep Understanding Questions:
- Do you understand what
loss.backward()actually does? Most engineers don't.- Can you debug when gradients vanish? You'll know why and how to fix it.
- Could you optimize a custom operation? You'll have built the primitives.
💡 The Learning Analogy:
Think of it like this: Pilots learn in small planes before flying 747s. You're learning the fundamentals that make you a better PyTorch engineer.
⚡ "How is this different from online tutorials that build neural networks?"
Most tutorials focus on isolated components - a Colab here, a notebook there. TinyTorch builds a fully integrated system.
🏗️ Systems Engineering Analogy:
Think of building a compiler or operating system. You don't just implement a lexer or a scheduler - you build how every component works together. Each piece must integrate seamlessly with the whole.📊 Component vs. System Approach:
Component Approach: Systems Approach (TinyTorch): ├── Build a neural network ├── Build a complete ML framework ├── Jupyter notebook demos ├── Full Python package with CLI ├── Isolated examples ├── Integrated: tensors → layers → training └── "Here's how ReLU works" ├── Production patterns: testing, profiling └── "Here's how EVERYTHING connects"🎯 Key Insight:
You learn systems engineering, not just individual algorithms. Like understanding how every part of a compiler interacts to turn code into executable programs.
💡 "Can't I just read papers/books instead of implementing?"
📚 Reading vs. 🔧 Building:
Reading about neural networks: Building neural networks: ├── "I understand the theory" ├── "Why are my gradients exploding?" ├── "Backprop makes sense" ├── "Oh, that's why we need gradient clipping" ├── "Adam is better than SGD" ├── "Now I see when each optimizer works" └── Theoretical knowledge └── Deep intuitive understanding🌟 The Reality Check:
Implementation forces you to confront reality - edge cases, numerical stability, memory management, performance trade-offs that papers gloss over.
🤔 "Isn't everything a Transformer now? Why learn old architectures?"
Great question! Transformers are indeed dominant, but they're built on the same foundations you'll implement:
🏗️ Transformer Building Blocks You'll Build:
- Attention is just matrix operations - which you'll build from tensors
- LayerNorm uses your activations and layers
- Adam optimizer powers Transformer training - you'll implement it
- Multi-head attention = your Linear layers + reshaping
🎯 The Strategic Reality:
Understanding foundations makes you the engineer who can optimize Transformers, not just use them. Plus, CNNs still power computer vision, RNNs drive real-time systems, and new architectures emerge constantly.
🎓 "I'm already good at ML. Is this too basic for me?"
🧪 Challenge Test - Can You:
- Implement Adam optimizer from the paper? (Not just use
torch.optim.Adam)- Explain why ReLU causes dying neurons and how to fix it?
- Debug a 50% accuracy drop after model deployment?
💪 Why Advanced Engineers Love TinyTorch:
It fills the "implementation gap" that most ML education skips. You'll go from understanding concepts to implementing production systems.
🧪 "Is this academic or practical?"
Both! TinyTorch bridges academic understanding with engineering reality:
🎓 Academic Rigor:
- Mathematical foundations implemented correctly
- Proper testing and validation methodologies
- Research-quality implementations you can trust
⚙️ Engineering Practicality:
- Production-style code organization and CLI tools
- Performance considerations and optimization techniques
- Real datasets, realistic scale, professional development workflow
⏰ "How much time does this take?"
📊 Time Investment: ~40-60 hours for complete framework
🎯 Flexible Learning Paths:
- Quick exploration: 1-2 modules to understand the approach
- Focused learning: Core modules (01-10) for solid foundations
- Complete mastery: All 16 modules for full framework expertise
✨ Self-Paced Design:
Each module is self-contained, so you can stop and start as needed.
🔄 "What if I get stuck or confused?"
🛡️ Built-in Support System:
- Progressive scaffolding: Each step builds on the previous, with guided implementations
- Comprehensive testing: 200+ tests ensure your code works correctly
- Rich documentation: Visual explanations, real-world context, debugging tips
- Professional error messages: Helpful feedback when things go wrong
- Modular design: Skip ahead or go back without breaking your progress
💡 Learning Philosophy:
The course is designed to guide you through complexity, not leave you struggling alone.
🚀 "What can I build after completing TinyTorch?"
🏗️ Your Framework Becomes the Foundation For:
- Research projects: Implement cutting-edge papers on solid foundations
- Specialized systems: Computer vision, NLP, robotics applications
- Performance engineering: GPU kernels, distributed training, quantization
- Custom architectures: New layer types, novel optimizers, experimental designs
🎯 Ultimate Skill Unlock:
You'll have the implementation skills to turn any ML paper into working code.