Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Support disaggregated prefill with Mooncake Transfer Engine #10884

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

[Core] Support disaggregated prefill with Mooncake Transfer Engine #10884

wants to merge 15 commits into from
+537 −3

Commits on Dec 2, 2024

  1. Rebase from main to work with PR 10502. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    d52dbc8 View commit details
    Browse the repository at this point in the history

  2. Update format of mooncake config ValueError. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    c8e9d07 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2024

  1. Modify metadata transfer logic to support tp. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    b718f1e View commit details
    Browse the repository at this point in the history

  2. Fix format to make ruff happy. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    08e2800 View commit details
    Browse the repository at this point in the history

  3. Add instructions when mooncake is not installed. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    8179746 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into upstream-mooncake-integration

    @ShangmingCai
    ShangmingCai committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    ba82d71 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2024

  1. Merge branch 'main' into upstream-mooncake-integration

    @ShangmingCai
    ShangmingCai committed Dec 4, 2024
    Configuration menu
    Copy the full SHA
    76d484c View commit details
    Browse the repository at this point in the history

  2. fix import order to make isort happy. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 4, 2024
    Configuration menu
    Copy the full SHA
    e912055 View commit details
    Browse the repository at this point in the history

  3. Fix format to make yapf happy. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 4, 2024
    Configuration menu
    Copy the full SHA
    2396f01 View commit details
    Browse the repository at this point in the history

  4. Add solution for ports conflict on the same node. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 4, 2024
    Configuration menu
    Copy the full SHA
    31514a0 View commit details
    Browse the repository at this point in the history

  5. Fix format to make mypy happy. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 4, 2024
    Configuration menu
    Copy the full SHA
    2ef10be View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2024

  1. Get head_size and num_heads from model config to address bugs on Volt… 

    …a and Turing GPUs.

Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 10, 2024
    Configuration menu
    Copy the full SHA
    0823e47 View commit details
    Browse the repository at this point in the history

  2. Add support for other metadata server backend. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 10, 2024
    Configuration menu
    Copy the full SHA
    6fb95fb View commit details
    Browse the repository at this point in the history

  3. Change code to align with PR 11058. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 10, 2024
    Configuration menu
    Copy the full SHA
    a5758b1 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2024

  1. Fix typo. 

    Signed-off-by: Shangming Cai <[email protected]>
    @ShangmingCai
    ShangmingCai committed Dec 11, 2024
    Configuration menu
    Copy the full SHA
    33e4455 View commit details
    Browse the repository at this point in the history