Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dropout #101

Draft
wants to merge 62 commits into
base: main_perf
Choose a base branch
from
Draft

Dropout #101

wants to merge 62 commits into from

Commits on Nov 15, 2024

  1. Alex's work 

    This is a combination of 11 commits.

save

fix: dropout=0.0 woorks

feat: dropout restrictions removed. failing tests

test: reduced tests to simple cases

test: failure is due to query + key padding mask NOT varlen itself

feat: varlen dropout fwd passes

fix: varlen bwd dropout works!

test: discovered  bwd error for non-dropout cases for large seqlen

save

save

use triton commit 3ca2f498e98ed7249b82722587c511a5610e00c4 -- now batched layout passes
    @alexkranias-amd @micmelesse
    alexkranias-amd authored and micmelesse committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    da57c2d View commit details
    Browse the repository at this point in the history

  2. pick test case

    @micmelesse
    micmelesse committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    b3151ef View commit details
    Browse the repository at this point in the history

  3. save philox offsets into metadata

    @micmelesse
    micmelesse committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    5ba7ed4 View commit details
    Browse the repository at this point in the history

  4. pass offset to ref

    @micmelesse
    micmelesse committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    7a0e919 View commit details
    Browse the repository at this point in the history

  5. common dropout mask

    @micmelesse
    micmelesse committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    00e5a34 View commit details
    Browse the repository at this point in the history

  6. simple droput out mask

    @micmelesse
    micmelesse committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    aec51ab View commit details
    Browse the repository at this point in the history

  7. start dropout ref. work on returning SD_Mask next with negative numbers

    @micmelesse
    micmelesse committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    5456df4 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2024

  1. refernce is working

    @micmelesse
    micmelesse committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    fe3cd4b View commit details
    Browse the repository at this point in the history

  2. dropout bwd ref faling case

    @micmelesse
    micmelesse committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    547cf0e View commit details
    Browse the repository at this point in the history

  3. transfer rng_state properly

    @micmelesse
    micmelesse committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    c7d917f View commit details
    Browse the repository at this point in the history

  4. save changes

    @micmelesse
    micmelesse committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    b8c3cca View commit details
    Browse the repository at this point in the history

  5. one dropout mask function

    @micmelesse
    micmelesse committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    708ecb4 View commit details
    Browse the repository at this point in the history

  6. save

    @micmelesse
    micmelesse committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    e977c25 View commit details
    Browse the repository at this point in the history

  7. save

    @micmelesse
    micmelesse committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    f6c8448 View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2024

  1. minizmize diff

    @micmelesse
    micmelesse committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    1410fbc View commit details
    Browse the repository at this point in the history

  2. save

    @micmelesse
    micmelesse committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    f233ac5 View commit details
    Browse the repository at this point in the history

  3. use torch.where in backward

    @micmelesse
    micmelesse committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    e6fc88e View commit details
    Browse the repository at this point in the history

  4. save

    @micmelesse
    micmelesse committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    9c162a2 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2024

  1. save

    @micmelesse
    micmelesse committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    52069bb View commit details
    Browse the repository at this point in the history

  2. save

    @micmelesse
    micmelesse committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    a1ccfeb View commit details
    Browse the repository at this point in the history

  3. dk works!

    @micmelesse
    micmelesse committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    1d47f2e View commit details
    Browse the repository at this point in the history

  4. passes

    @micmelesse
    micmelesse committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    458419a View commit details
    Browse the repository at this point in the history

  5. reference is working. TODO" attn_ref is broken

    @micmelesse
    micmelesse committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    7a4eafe View commit details
    Browse the repository at this point in the history

  6. varlen ref working

    @micmelesse
    micmelesse committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    a247186 View commit details
    Browse the repository at this point in the history

  7. attn failing case

    @micmelesse
    micmelesse committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    c59b7b8 View commit details
    Browse the repository at this point in the history

  8. with ones. attn_ref matches. fails with randn. we are seeing failure … 

    …with large sizes from dv.
    @micmelesse
    micmelesse committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    ea2d634 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. save

    @micmelesse
    micmelesse committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    24930e3 View commit details
    Browse the repository at this point in the history

  2. skip attn matrices

    @micmelesse
    micmelesse committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    42735d0 View commit details
    Browse the repository at this point in the history

  3. compare the masks and find failing case

    @micmelesse
    micmelesse committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    5c5a5d6 View commit details
    Browse the repository at this point in the history

  4. rm cdiv_fn

    @micmelesse
    micmelesse committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    416087d View commit details
    Browse the repository at this point in the history

  5. put dropout and alibi in common

    @micmelesse
    micmelesse committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    fb3f0b3 View commit details
    Browse the repository at this point in the history

  6. save

    @micmelesse
    micmelesse committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    8810582 View commit details
    Browse the repository at this point in the history

  7. compare masks

    @micmelesse
    micmelesse committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    998a732 View commit details
    Browse the repository at this point in the history

  8. save

    @micmelesse
    micmelesse committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    166bb86 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. save

    @micmelesse
    micmelesse committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    1b9e085 View commit details
    Browse the repository at this point in the history

  2. pytorch ref is using tiles

    @micmelesse
    micmelesse committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    151f294 View commit details
    Browse the repository at this point in the history

  3. save

    @micmelesse
    micmelesse committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    3ce0cc4 View commit details
    Browse the repository at this point in the history

  4. save

    @micmelesse
    micmelesse committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    77f356e View commit details
    Browse the repository at this point in the history

  5. tl_rand_ref

    @micmelesse
    micmelesse committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    d038863 View commit details
    Browse the repository at this point in the history

  6. cache ref dropout mask

    @micmelesse
    micmelesse committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    1520b9e View commit details
    Browse the repository at this point in the history

  7. new generate_dropout_mask_ref using tiling

    @micmelesse
    micmelesse committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    4ce02b8 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2024

  1. issolate failing varlen case

    @micmelesse
    micmelesse committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    4fef4b3 View commit details
    Browse the repository at this point in the history

  2. simple dropout

    @micmelesse
    micmelesse committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    d4155b3 View commit details
    Browse the repository at this point in the history

  3. loop on k

    @micmelesse
    micmelesse committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    18ae50a View commit details
    Browse the repository at this point in the history

  4. print rng_outputs

    @micmelesse
    micmelesse committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    4c8f68d View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2024

  1. save

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    473f475 View commit details
    Browse the repository at this point in the history

  2. fwd kernel works

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    cbc8faf View commit details
    Browse the repository at this point in the history

  3. save

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    07b0e34 View commit details
    Browse the repository at this point in the history

  4. dv passed

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    1425cfe View commit details
    Browse the repository at this point in the history

  5. close to dk

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    a0ab26f View commit details
    Browse the repository at this point in the history

  6. simple ref

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    d802208 View commit details
    Browse the repository at this point in the history

  7. save

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    eabfb85 View commit details
    Browse the repository at this point in the history

  8. seperate droped and scaled in ref and triton kernel

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    43d5796 View commit details
    Browse the repository at this point in the history

  9. ref changes

    @micmelesse
    micmelesse committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    5651fee View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2024

  1. working delta with dp

    @micmelesse
    micmelesse committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    a2795be View commit details
    Browse the repository at this point in the history

  2. find failing dv failures

    @micmelesse
    micmelesse committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    474ef3a View commit details
    Browse the repository at this point in the history

  3. find failing case due to delta

    @micmelesse
    micmelesse committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    0f737d5 View commit details
    Browse the repository at this point in the history

  4. save

    @micmelesse
    micmelesse committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    90609d8 View commit details
    Browse the repository at this point in the history

  5. delta from dp working

    @micmelesse
    micmelesse committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    7720621 View commit details
    Browse the repository at this point in the history

  6. bwd impl green

    @micmelesse
    micmelesse committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    0d78d4a View commit details
    Browse the repository at this point in the history

  7. enable test fwd

    @micmelesse
    micmelesse committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    aa4275a View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2024

  1. save

    @micmelesse
    micmelesse committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    c0e4902 View commit details
    Browse the repository at this point in the history