Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multithreaded replication WIP #1454

Draft
wants to merge 55 commits into
base: master
Choose a base branch
from
Draft

Multithreaded replication WIP #1454

wants to merge 55 commits into from

Commits on Sep 30, 2024

  1. WIP.

    @arthurschreiber
    arthurschreiber committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    37c1abd View commit details
    Browse the repository at this point in the history

  2. Fixes.

    @arthurschreiber
    arthurschreiber committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    5d69fd9 View commit details
    Browse the repository at this point in the history

  3. Fixups.

    @arthurschreiber
    arthurschreiber committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    df29aa8 View commit details
    Browse the repository at this point in the history

  4. Special handling for first job.

    @arthurschreiber
    arthurschreiber committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    81a6e6e View commit details
    Browse the repository at this point in the history

  5. Fix deadlocks.

    @arthurschreiber
    arthurschreiber committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    b44c6df View commit details
    Browse the repository at this point in the history

  6. Update docs and simulate changes.

    @arthurschreiber
    arthurschreiber committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    55ce5a2 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. start StreamTransaction implementation

    @meiji163
    meiji163 committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    0d8f39b View commit details
    Browse the repository at this point in the history

  2. add job Coordinator

    @meiji163
    meiji163 committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    22a4b8c View commit details
    Browse the repository at this point in the history

  3. fix job complete logic

    @meiji163
    meiji163 committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    3fd6e71 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. add docker compose

    @meiji163
    meiji163 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    f209592 View commit details
    Browse the repository at this point in the history

  2. fix application of applyEvent.writeFunc

    @meiji163
    meiji163 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    c6c877e View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Send off transaction events as soon as we see a GTID event. 

    Multiple `TableMap` events can happen inside of a single transaction, so we can't really use them to filter out events early or anything like that. Instead, we'll just skip them.

Also adds some basic test cases for the transaction streaming.
    @arthurschreiber
    arthurschreiber committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    b031166 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. WIP. 

    Co-authored-by: Daniel Joos <[email protected]>
    @arthurschreiber @danieljoos
    arthurschreiber and danieljoos committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    f81a790 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. WIP.

    @arthurschreiber
    arthurschreiber committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    e3b2cda View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. fix MarkTransactionCompleted

    @meiji163
    meiji163 committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    3ba9058 View commit details
    Browse the repository at this point in the history

  2. go mod tidy

    @meiji163
    meiji163 committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    74f6c9c View commit details
    Browse the repository at this point in the history

  3. configure max idle connections

    @meiji163
    meiji163 committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    eddc1c9 View commit details
    Browse the repository at this point in the history

  4. vendor packages

    @meiji163
    meiji163 committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    6321e73 View commit details
    Browse the repository at this point in the history

  5. track binlog coords

    @meiji163
    meiji163 committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    5600b91 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. binlog streamer reconnect

    @meiji163
    meiji163 committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    3f47ebd View commit details
    Browse the repository at this point in the history

  2. fix TestMigrate

    @meiji163
    meiji163 committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    e81aabf View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. worker-stats command

    @meiji163
    meiji163 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    e4da5f8 View commit details
    Browse the repository at this point in the history

  2. setup mysql in CI

    @meiji163
    meiji163 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    c5e239c View commit details
    Browse the repository at this point in the history

  3. set transaction_write_set_extraction in test

    @meiji163
    meiji163 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    2e78f6f View commit details
    Browse the repository at this point in the history

  4. add ci mysql opt flags

    @meiji163
    meiji163 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    875d00d View commit details
    Browse the repository at this point in the history

  5. Revert "set transaction_write_set_extraction in test" 

    This reverts commit 2e78f6f.
    @meiji163
    meiji163 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    9a022e2 View commit details
    Browse the repository at this point in the history

  6. change ci mysql opts

    @meiji163
    meiji163 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    18dcee1 View commit details
    Browse the repository at this point in the history

  7. try custom docker run

    @meiji163
    meiji163 committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    299df37 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Make the linter happy.

    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    941689f View commit details
    Browse the repository at this point in the history

  2. Use testcontainers.

    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    9e3bc1c View commit details
    Browse the repository at this point in the history

  3. WIP.

    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    113e674 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' of https://github.com/github/gh-ost into meiji1… 

    …63/parallel-repl
    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    8e38b86 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' of https://github.com/github/gh-ost into meiji1… 

    …63/parallel-repl
    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d7ccab9 View commit details
    Browse the repository at this point in the history

  6. fix 2 tests, TestMigrate still not working

    @meiji163
    meiji163 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    06c7082 View commit details
    Browse the repository at this point in the history

  7. Fix applier tests.

    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d7dc97b View commit details
    Browse the repository at this point in the history

  8. Fix migrator tests.

    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    bcc7e7a View commit details
    Browse the repository at this point in the history

  9. Add error assertions.

    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    b747d98 View commit details
    Browse the repository at this point in the history

  10. 🔥

    @arthurschreiber
    arthurschreiber committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    1a5be0b View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Fix applier connection pool size.

    @arthurschreiber
    arthurschreiber committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    1942455 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Merge branch 'master' into meiji163/parallel-repl

    @arthurschreiber
    arthurschreiber authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    85cab4d View commit details
    Browse the repository at this point in the history

  2. Fix merge conflict.

    @arthurschreiber
    arthurschreiber committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    2ba0cb2 View commit details
    Browse the repository at this point in the history

  3. Prepare queries.

    @arthurschreiber
    arthurschreiber committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    b82e8f9 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. pass throttler to Coordinator

    @meiji163
    meiji163 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    fa7c484 View commit details
    Browse the repository at this point in the history

  2. track time waiting on event channels

    @meiji163
    meiji163 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    126c981 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. add flag for number of Coordinator workers

    @meiji163
    meiji163 committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    641fe92 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/github/gh-ost into meiji1… 

    …63/parallel-repl
    @arthurschreiber
    arthurschreiber committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    615d1df View commit details
    Browse the repository at this point in the history

  3. Fix test case.

    @arthurschreiber
    arthurschreiber committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    0a8787e View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2024

  1. Merge branch 'master' into meiji163/parallel-repl

    @meiji163
    meiji163 authored Nov 18, 2024
    Configuration menu
    Copy the full SHA
    6aaa374 View commit details
    Browse the repository at this point in the history

  2. remove unused streamer

    @meiji163
    meiji163 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    f6ec835 View commit details
    Browse the repository at this point in the history

  3. fix coordinator test

    @meiji163
    meiji163 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    0555d72 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into meiji163/parallel-repl

    @meiji163
    meiji163 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    569d035 View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2024

  1. fix Coordinator test

    @meiji163
    meiji163 committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    9b5e2bc View commit details
    Browse the repository at this point in the history

  2. fix migrator test

    @meiji163
    meiji163 committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    6689aeb View commit details
    Browse the repository at this point in the history

  3. linter fix

    @meiji163
    meiji163 committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    53e953d View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. Merge branch 'master' into meiji163/parallel-repl

    @meiji163
    meiji163 authored Nov 22, 2024
    Configuration menu
    Copy the full SHA
    e1ca9cd View commit details
    Browse the repository at this point in the history