Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model per instance model-mesh by default #108

Open
fsatka opened this issue Sep 25, 2023 · 1 comment
Open

Model per instance model-mesh by default #108

fsatka opened this issue Sep 25, 2023 · 1 comment
Labels
question Further information is requested

Comments

@fsatka
Copy link

fsatka commented Sep 25, 2023

Now model load only on one instance, and lazy loading on another pods, when reauest has come.

Can we modify internal modelmesh parameters for default loading model on all ServingRuntime instances?

@ckadner ckadner added the question Further information is requested label Jan 19, 2024
@ckadner
Copy link
Member

ckadner commented Jan 19, 2024

@fsatka -- ModelMesh was designed to optimize resource utilization. Why would you want to load additional instances of the same model/predictor/ISVC on all serving runtime pods regardless of inference request traffic? Just for testing purposes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants