Skip to content

MoE v1.0.0

Latest
Compare
Choose a tag to compare
@RissyRan RissyRan released this 10 Sep 06:36
· 248 commits to main since this release

MoE v1.0.0 supports:

  • Megablox with Fully Sharded Data Parallelism (FSDP) and Token Parallelism (TP)
  • Dropping strategies with FSDP, TP, and Expert Parallelism (EP)