Code

Marlin MoE integration#2

Code

Closed

ElizaWszola wants to merge 111 commits intomainneuralmagic/vllm:mainfrom marlin-moe-integrationneuralmagic/vllm:marlin-moe-integrationCopy head branch name to clipboard

+1,370-335

Commits

Commits on Aug 2, 2024

• Moving branch to a different repo

  ElizaWszola

  committed
  5a2ab25

  Copy full SHA for 5a2ab25

• clean up the CPU code

  ElizaWszola

  committed
  b39dba4

  Copy full SHA for b39dba4

• Fix build issues

  ElizaWszola

  committed
  b0c4671

  Copy full SHA for b0c4671

Commits on Aug 7, 2024

• Refactoring for maintainability

  DhruvaBansal00

  committed
  e5c1a81

  Copy full SHA for e5c1a81

• Fixing tests

  DhruvaBansal00

  committed
  7da678e

  Copy full SHA for 7da678e

Commits on Aug 8, 2024

• Addressing repacking comment

  DhruvaBansal00

  committed
  641696b

  Copy full SHA for 641696b

• gptq -> marlin renaming

  DhruvaBansal00

  committed
  3cef667

  Copy full SHA for 3cef667

• Undo formatting changes

  DhruvaBansal00

  committed
  a6710af

  Copy full SHA for a6710af

• Final formatting change

  DhruvaBansal00

  committed
  e29107f

  Copy full SHA for e29107f

Commits on Aug 12, 2024

• Switching to mixtral file for quantized mixtral

  DhruvaBansal00

  committed
  099d61e

  Copy full SHA for 099d61e

• Bug fixes

  DhruvaBansal00

  committed
  bdf6bdc

  Copy full SHA for bdf6bdc

• is quantized change

  DhruvaBansal00

  committed
  19c5c59

  Copy full SHA for 19c5c59

• debug stat

  DhruvaBansal00

  committed
  3b7cc60

  Copy full SHA for 3b7cc60

• replace wiehgt name with param name

  DhruvaBansal00

  committed
  d2c4754

  Copy full SHA for d2c4754

• typo

  DhruvaBansal00

  committed
  f579cb2

  Copy full SHA for f579cb2

• debug

  DhruvaBansal00

  committed
  79394eb

  Copy full SHA for 79394eb

• more debug

  DhruvaBansal00

  committed
  ec75f4e

  Copy full SHA for ec75f4e

• only relevant logging

  DhruvaBansal00

  committed
  91ca970

  Copy full SHA for 91ca970

• log

  DhruvaBansal00

  committed
  1b9d5bb

  Copy full SHA for 1b9d5bb

• log

  DhruvaBansal00

  committed
  ec06719

  Copy full SHA for ec06719

• removing qzero weights

  DhruvaBansal00

  committed
  71d82e1

  Copy full SHA for 71d82e1

• Qzeors in expert mapping

  DhruvaBansal00

  committed
  d3465d0

  Copy full SHA for d3465d0

• Debug

  DhruvaBansal00

  committed
  226ee26

  Copy full SHA for 226ee26

• Load qzero

  DhruvaBansal00

  committed
  21d7d27

  Copy full SHA for 21d7d27

• rm 2x

  DhruvaBansal00

  committed
  2dabb4b

  Copy full SHA for 2dabb4b

• Mapping for scales

  DhruvaBansal00

  committed
  6366976

  Copy full SHA for 6366976

• rm logging

  DhruvaBansal00

  committed
  d63c096

  Copy full SHA for d63c096

• Adding lyaer wise logging

  DhruvaBansal00

  committed
  360fef4

  Copy full SHA for 360fef4

• shard ids

  DhruvaBansal00

  committed
  c23d616

  Copy full SHA for c23d616

• Loading qzero correctly

  DhruvaBansal00

  committed
  8d81d14

  Copy full SHA for 8d81d14

• List operand

  DhruvaBansal00

  committed
  22e1aa7

  Copy full SHA for 22e1aa7

• If clause

  DhruvaBansal00

  committed
  81e01f3

  Copy full SHA for 81e01f3

• Able to load layers

  DhruvaBansal00

  committed
  dcfd32d

  Copy full SHA for dcfd32d

• Setting load quant to false

  DhruvaBansal00

  committed
  f04cbea

  Copy full SHA for f04cbea

• Disabling logging

  DhruvaBansal00

  committed
  a56821d

  Copy full SHA for a56821d

Commits on Aug 13, 2024

• Removing *2 in marlin moe repack

  DhruvaBansal00

  committed
  7f961c6

  Copy full SHA for 7f961c6

• *4 in marlin moe repack

  DhruvaBansal00

  committed
  4a6c7ff

  Copy full SHA for 4a6c7ff

• bits

  DhruvaBansal00

  committed
  e6cd286

  Copy full SHA for e6cd286

• *4

  DhruvaBansal00

  committed
  90241c4

  Copy full SHA for 90241c4

• intermediate size

  DhruvaBansal00

  committed
  67409e9

  Copy full SHA for 67409e9

• repeat keyword

  DhruvaBansal00

  committed
  539032e

  Copy full SHA for 539032e

• hidden size

  DhruvaBansal00

  committed
  57b1cbe

  Copy full SHA for 57b1cbe

• intermediate size back

  DhruvaBansal00

  committed
  87f1dd4

  Copy full SHA for 87f1dd4

• permute scales w3

  DhruvaBansal00

  committed
  4c073c2

  Copy full SHA for 4c073c2

• *2

  DhruvaBansal00

  committed
  d732493

  Copy full SHA for d732493

• log

  DhruvaBansal00

  committed
  fdc22c4

  Copy full SHA for fdc22c4

• shape as 2

  DhruvaBansal00

  committed
  272822e

  Copy full SHA for 272822e

• test

  DhruvaBansal00

  committed
  3ce045e

  Copy full SHA for 3ce045e

• Increasing to 4 and changing assert

  DhruvaBansal00

  committed
  c4ba477

  Copy full SHA for c4ba477

• logging

  DhruvaBansal00

  committed
  2ea8370

  Copy full SHA for 2ea8370

• marlin moe repack change

  DhruvaBansal00

  committed
  8287025

  Copy full SHA for 8287025

• mult qweight shape by pack factor

  DhruvaBansal00

  committed
  53b23b9

  Copy full SHA for 53b23b9

• Potential support for 8 bit

  DhruvaBansal00

  committed
  bc40786

  Copy full SHA for bc40786

• undo change

  DhruvaBansal00

  committed
  bea13de

  Copy full SHA for bea13de

• qzeros

  DhruvaBansal00

  committed
  a3a9114

  Copy full SHA for a3a9114

• switching traffic to mixtral quant

  DhruvaBansal00

  committed
  eb916f9

  Copy full SHA for eb916f9

• compat

  DhruvaBansal00

  committed
  017d6f8

  Copy full SHA for 017d6f8

• Passing intermediate tensor into mixtral in quant file

  DhruvaBansal00

  committed
  eb9c087

  Copy full SHA for eb9c087

• Removing intemediate tensors from forward

  DhruvaBansal00

  committed
  ea3cf18

  Copy full SHA for ea3cf18

• load weights from quant

  DhruvaBansal00

  committed
  4f6b4ca

  Copy full SHA for 4f6b4ca

• Mixtral load weights change:

  DhruvaBansal00

  committed
  7ec27d9

  Copy full SHA for 7ec27d9

• none shard id change

  DhruvaBansal00

  committed
  aa1fe77

  Copy full SHA for aa1fe77

Commits on Aug 15, 2024

• Use class from mixtral_quant

  DhruvaBansal00

  committed
  ae8fb15

  Copy full SHA for ae8fb15

• Removing lora from mixtral model init

  DhruvaBansal00

  committed
  b863981

  Copy full SHA for b863981

• Adding empty intermediate tensors

  DhruvaBansal00

  committed
  5556d28

  Copy full SHA for 5556d28

• Building quantMixtralModel

  DhruvaBansal00

  committed
  c484a37

  Copy full SHA for c484a37

• fused moe test

  DhruvaBansal00

  committed
  0344e72

  Copy full SHA for 0344e72

• Lora enabled mixtral

  DhruvaBansal00

  committed
  8c8b3fa

  Copy full SHA for 8c8b3fa

• LoRAMixtralModel compat

  DhruvaBansal00

  committed
  dff59cd

  Copy full SHA for dff59cd

• remove prefix

  DhruvaBansal00

  committed
  33f7e51

  Copy full SHA for 33f7e51

• use fused moe

  DhruvaBansal00

  committed
  fdba917

  Copy full SHA for fdba917

• remove org num embeddings

  DhruvaBansal00

  committed
  780471e

  Copy full SHA for 780471e

• pass use fused moe into decoder

  DhruvaBansal00

  committed
  c0970f1

  Copy full SHA for c0970f1

• Mixtral for causal lm load func

  DhruvaBansal00

  committed
  6a1a838

  Copy full SHA for 6a1a838

• Copying over quant mixtral

  DhruvaBansal00

  committed
  5c3e857

  Copy full SHA for 5c3e857

• Passing prefix

  DhruvaBansal00

  committed
  8d327de

  Copy full SHA for 8d327de

• Weight load

  DhruvaBansal00

  committed
  d337aea

  Copy full SHA for d337aea

• Weight load back

  DhruvaBansal00

  committed
  379f3e8

  Copy full SHA for 379f3e8

• Load with name not weight name

  DhruvaBansal00

  committed
  a5d356e

  Copy full SHA for a5d356e

• params dict should load from old name

  DhruvaBansal00

  committed
  62c0135

  Copy full SHA for 62c0135

• logging name and parmas

  DhruvaBansal00

  committed
  d23c00c

  Copy full SHA for d23c00c

• log expert parmas map

  DhruvaBansal00

  committed
  6dda447

  Copy full SHA for 6dda447

• parity with prev commits

  DhruvaBansal00

  committed
  67ce7b6

  Copy full SHA for 67ce7b6

• Adding qzeros to mapping

  DhruvaBansal00

  committed
  bd933c9

  Copy full SHA for bd933c9

• Remove log

  DhruvaBansal00

  committed
  77cd095

  Copy full SHA for 77cd095

• Remove is quantized

  DhruvaBansal00

  committed
  529191e

  Copy full SHA for 529191e

• Assume fused true

  DhruvaBansal00

  committed
  2450543

  Copy full SHA for 2450543

• rm fused true

  DhruvaBansal00

  committed
  8cba45e

  Copy full SHA for 8cba45e

• Switching to mixtral moe

  DhruvaBansal00

  committed
  10940a5

  Copy full SHA for 10940a5

• Precision changes

  DhruvaBansal00

  committed
  895ffbe

  Copy full SHA for 895ffbe

• Cleanup

  DhruvaBansal00

  committed
  e54b2e4

  Copy full SHA for e54b2e4

• Mixtral quant parity:

  DhruvaBansal00

  committed
  b4f23dc

  Copy full SHA for b4f23dc

• fixing tests

  DhruvaBansal00

  committed
  d59fe3b

  Copy full SHA for d59fe3b

• Tests working and correctness verified

  DhruvaBansal00

  committed
  0d9cbdc

  Copy full SHA for 0d9cbdc

• Formating

  DhruvaBansal00

  committed
  112aa40

  Copy full SHA for 112aa40

Commits on Aug 19, 2024

• Moving single marlin alongside fused marlin

  DhruvaBansal00

  committed
  1ca9098

  Copy full SHA for 1ca9098

• Removing unused imports

  DhruvaBansal00

  committed
  4d41425

  Copy full SHA for 4d41425

• single marlin moe import

  DhruvaBansal00

  committed
  4907f43

  Copy full SHA for 4907f43

Commits on Aug 20, 2024

• Merge branch 'main' into marlin-moe-integration

  ElizaWszola

  committed
  8f4648c

  Copy full SHA for 8f4648c

• Merge branch 'marlin-moe-integration' into gptq-marlin-refactor

  ElizaWszola

  committed
  8225037

  Copy full SHA for 8225037

Commits on Aug 21, 2024

• Unify shard_id to be of str w[1-3] format

  ElizaWszola

  committed
  315e3b6

  Copy full SHA for 315e3b6

Commits on Aug 22, 2024

• Merge pull request #4 from DhruvaBansal00/gptq-marlin-refactor

  Show description for 34bb5b0
  ElizaWszola

  authored
  34bb5b0

  Copy full SHA for 34bb5b0

• Merge branch 'main' into marlin-moe-integration

  ElizaWszola

  committed
  fd4bb21

  Copy full SHA for fd4bb21

Commits on Aug 26, 2024

• Unfused codepath for non-supported quant_types

  ElizaWszola

  committed
  7956a69

  Copy full SHA for 7956a69

Commits on Aug 28, 2024

• uint8b128 support

  ElizaWszola

  committed
  2511f78

  Copy full SHA for 2511f78

Commits on Aug 29, 2024

• Merge branch 'main' into marlin-moe-integration

  ElizaWszola

  committed
  f875842

  Copy full SHA for f875842

• Cleanup, compressed tensors compatibility

  ElizaWszola

  committed
  d8feb8d

  Copy full SHA for d8feb8d

• update todo

  ElizaWszola

  committed
  3676621

  Copy full SHA for 3676621

Commits on Aug 30, 2024

• Fix merge

  ElizaWszola

  committed
  75e3dd5

  Copy full SHA for 75e3dd5

• bad paste

  ElizaWszola

  committed
  a5f5a74

  Copy full SHA for a5f5a74

Commits on Sep 2, 2024

• GPTQFusedMoE layer

  ElizaWszola

  committed
  e305306

  Copy full SHA for e305306