Code

Add MultiHeadAttention layer#392

Code

Merged

Dobiasd merged 38 commits intomasterDobiasd/frugally-deep:masterfrom multi-head-attentionDobiasd/frugally-deep:multi-head-attentionCopy head branch name to clipboardDec 31, 2023

+228-2

Commits

Commits on Aug 14, 2023

• Add MultiHeadAttention layer

  Dobiasd

  committed
  a2992f9

• Pass parameters to multi_head_attention_layer

  Dobiasd

  committed
  e0a886d

• Persist weight shapes too

  Dobiasd

  committed
  931bc27

Commits on Sep 12, 2023

• Merge branch 'master' into multi-head-attention

  Dobiasd

  committed
  3a5f8ec

Commits on Sep 13, 2023

• Merge branch 'master' into multi-head-attention

  Dobiasd

  committed
  a1e0a58

Commits on Nov 1, 2023

• Merge master

  Dobiasd

  committed
  60f77d5

• Add comment with link to stackoverflow question

  Dobiasd

  committed
  14aa03a

Commits on Nov 7, 2023

• Implement special treatment for the inbound_nodes format of MultiHeadAttention

  Dobiasd

  committed
  e8efb24

• Adjust comment in multi_head_attention_layer

  Dobiasd

  committed
  7a7928d

Commits on Nov 28, 2023

• Merge branch 'master' into multi-head-attention

  Dobiasd

  committed
  84c492e

Commits on Dec 16, 2023

• comment

  Dobiasd

  committed
  3bfbf07

• Merge branch 'master' into multi-head-attention

  Dobiasd

  committed
  1b6597e

Commits on Dec 26, 2023

• add comment

  Dobiasd

  committed
  9743a49

• Merge branch 'master' into multi-head-attention

  Dobiasd

  committed
  9231a60

• Merge branch 'master' into multi-head-attention

  Dobiasd

  committed
  78c0d4e

• Convert weights to tensors for ctor

  Dobiasd

  committed
  09d5868

• Separate weights and biases

  Dobiasd

  committed
  212d609

Commits on Dec 29, 2023

• apply dense layers to query, value and key

  Dobiasd

  committed
  118f663

• only allow the usual shapes

  Dobiasd

  committed
  53f2d9b

• adjust tests

  Dobiasd

  committed
  4184515

Commits on Dec 30, 2023

• Fix default bias and loading order of key and value weights

  Dobiasd

  committed
  291e127

• fix shapes, add tests

  Dobiasd

  committed
  5b3fbd2

Commits on Dec 31, 2023

• decode weights into multiple heads

  Dobiasd

  committed
  0e17267

• enable tests

  Dobiasd

  committed
  d896c4c

• create dense output layer separately

  Dobiasd

  committed
  3176f89

• clean up

  Dobiasd

  committed
  c7adc7c

• add more tests

  Dobiasd

  committed
  db62540

• shorten

  Dobiasd

  committed
  702cb60

• const

  Dobiasd

  committed
  b825b57

• teeeests

  Dobiasd

  committed
  67264ea

• fix the distribution calculation by dividing with the square root of query_size before applying the softmax

  Dobiasd

  committed
  8f630a8

• remove debug tests

  Dobiasd

  committed
  0c6cc0a

• remove todo comment

  Dobiasd

  committed
  0d2be86

• Check for attention_axes=None in conversion

  Dobiasd

  committed
  fd6e7c4

• Do not pass unused attention_axes

  Dobiasd

  committed
  a95abf4

• double-check weights shapes

  Dobiasd

  committed
  41ac53a

• Revert debug output

  Dobiasd

  committed
  b35deb9

• Add MultiHeadAttention layer to list of supported layers in README

  Dobiasd

  committed
  7a574b7