[Fix] - Router: add model_name index for O(1) deployment lookups #15113
+106
−17
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add model_name_to_deployment_indices mapping to optimize _get_all_deployments() from O(n) to O(1) + O(k) lookups. - Add model_name_to_deployment_indices: Dict[str, List[int]] - Add _build_model_name_index() to build/maintain the index - Update _add_model_to_list_and_index_map() to maintain both indices - Refactor to use idx = len(self.model_list) before append (cleaner) - Optimize _get_all_deployments() to use index instead of linear scan
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Add single comprehensive test for _build_model_name_index() function to fix code coverage CI failure. The test verifies: - Index correctly maps model_name to deployment indices - Handles multiple deployments per model_name - Clears and rebuilds index correctly Fixes: CI code coverage error for _build_model_name_index
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
[Fix] - Router: add model_name index for O(1) deployment lookups
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitType
🧹 Refactoring
Changes
Performance Gains
Summary
Average, min and rps changed the most.
Before
After