fix free cache in MPEngine branch #3670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

JimyMa wants to merge 44 commits into InternLM:main from JimyMa:mp-engine-fix-free-cache

Contributor

JimyMa commented Jun 25, 2025 •

edited

Loading

send free signal in decode engine for release cache immediately when migration done
a series of fault tolerance

grimoire and others added 16 commits

June 8, 2025 19:36


          mp router

3099c8f


          optimize

1bdacbf


          better streaming

4691d74


          optimize streaming

b8b5b3b


          Merge branch 'main' into mp-engine

a7476c3


          close engine

362240a


          safe exit

899da60


          support pd

d49146a


          merge main

a1e92a1


          fix loader

0be4fc8


          optimize

5f2939e


          Merge branch 'main' into mp-engine

f428506


          safe exit

3285cc6


          safe exit

3ff8d28


          send free signal in decode engine for release cache immediately when …

c3cc58a

…migration done


          typo bugfix

039f74a

lvhan028 requested a review from grimoire

June 26, 2025 13:12

Collaborator

grimoire commented Jul 2, 2025

Please solve the comflicts

JimyMa added 5 commits

July 9, 2025 22:53


          Merge remote-tracking branch 'origin/main' into mp-engine-fix-free-cache

f1f1430


          fix lint

4e83e63


          adapt new data structure to mooncake

76ccf2b


          revert useless change

cde1145


          revert useless change

95914fd

grimoire reviewed

View reviewed changes

lmdeploy/pytorch/engine/mp_engine/mp_engine.py Outdated Show resolved Hide resolved

grimoire reviewed

View reviewed changes

lmdeploy/pytorch/models/deepseek_v2.py Outdated Show resolved Hide resolved

grimoire reviewed

View reviewed changes

lmdeploy/pytorch/paging/scheduler.py Outdated Show resolved Hide resolved

grimoire reviewed

View reviewed changes

requirements/serve.txt Outdated Show resolved Hide resolved

grimoire reviewed

View reviewed changes

lmdeploy/pytorch/engine/engine.py Show resolved Hide resolved

JimyMa added 2 commits

July 10, 2025 22:09


          disable fake eplb by default

fade092


          add engine remove fault tolerance

5785c54

JimyMa added 13 commits

July 14, 2025 14:28


          add p2p disconnection logic

36a49e3


          pass lint check

5537f7c


          Merge remote-tracking branch 'origin/main' into mp-engine-fix-free-cache

123010c


          handle review

31326cc


          handle review

0fb9a01


          avoid circle import

4ad188a


          add __init__ file

1ef362e


          not expose fields of engine in mp_engine

92ca4d4


          bugfix and enable p2p disconnection

20f6578


          bugfix of fault tolerance

b0f67cf


          add gc interface and bugfix

d96d36a


          handle connection error and no pd url response in proxy

c83c329


          bugfix of freecache and drop connect request

c1abc70

grimoire reviewed

View reviewed changes

lmdeploy/pytorch/paging/scheduler.py Outdated Show resolved Hide resolved

JimyMa added 2 commits

July 14, 2025 21:46


          remove logger.debug in scheduler

96f0302


          remove logger.debug in engine_conn

c2a3abe

grimoire reviewed

View reviewed changes

lmdeploy/pytorch/engine/mp_engine/mp_engine.py

               from lmdeploy.utils import get_logger
               logger = get_logger('lmdeploy')
+              if TYPE_CHECKING:

Collaborator

grimoire Jul 15, 2025

Why do we need this branch?

Contributor Author

JimyMa Jul 15, 2025

Here, creating a branch and using TYPE_CHECKING to import the Engine module avoids circular references while still enabling IDE highlighting and navigation.

grimoire approved these changes

View reviewed changes

Collaborator

grimoire left a comment

LGTM

JimyMa and others added 6 commits

July 15, 2025 23:40


          Merge branch 'InternLM:main' into mp-engine-fix-free-cache

1c71f7e


          remove redundant file

cb1aa04


          error => warning

2d617c0


          fix typo

98a7d9f


          1. add dummy prefill function in proxy and engine

3819a4b


          1. dummy prefill bug fix

grimoire mentioned this pull request

add remote logs;optimize forward lock #3737

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet