fix: Preserve Bedrock inference profile IDs in health checks #15947
+144
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fixes issue where health checks were stripping inference profile IDs - Preserves cross-region inference profile prefixes (us., eu., apac., jp., au., us-gov., global.) - Strips only AWS region routing while preserving routes and handlers - Resolves both issue BerriAI#15807 and inference profile requirement errors - Adds comprehensive tests for all Bedrock model format combinations Issue BerriAI#15807 attempted to fix regional Bedrock model health checks but was too aggressive, stripping cross-region inference profile prefixes that AWS requires. This caused errors: "Invocation of model ID X with on-demand throughput isn't supported. Retry your request with the ID or ARN of an inference profile." The fix now correctly: - Strips AWS regions (us-west-2, eu-central-1, etc.) from routing - Preserves CRIS prefixes (us., eu., etc.) required by AWS - Preserves routes (converse/, invoke/) - Preserves handlers (llama/, deepseek_r1/) - Only affects Bedrock models (checked via startswith) Test coverage includes 20+ scenarios for all Bedrock model format combinations.
|
@ylgibby is attempting to deploy a commit to the CLERKIEAI Team on Vercel. A member of the Team first needs to authorize it. |
|
|
ylgibby
force-pushed
the
fix/bedrock-health-check-inference-profiles
branch
from
ea30f23 to
1f88d71
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes #15949
In v1.79.0-stable, all Bedrock health checks are failing when using inference profile IDs. This PR fixes the issue while maintaining the original fix for #15807.
Root Cause
PR #15808 attempted to fix issue #15807 (regional routing in health checks) by using
get_base_model()to strip region prefixes. However, this was too aggressive and also stripped AWS-required Cross-Region Inference Profile (CRIS) prefixes likeus.,eu.,apac., etc.Solution
This PR implements a more targeted approach that:
us-west-2,eu-central-1)us.,eu.,apac.)converse/,invoke/)llama/,deepseek_r1/)Implementation
Modified
_update_litellm_params_for_health_check()inlitellm/proxy/health_check.pyto:BedrockModelInfo.all_global_regionsExamples
Regional routing (issue #15807) - strips region ✅
bedrock/us-west-2/anthropic.claude-3-5-sonnet-20240620-v1:0anthropic.claude-3-5-sonnet-20240620-v1:0Inference profiles - preserves CRIS prefix ✅
bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0us.anthropic.claude-3-5-sonnet-20240620-v1:0Complex routing - preserves route + strips region ✅
bedrock/converse/us-west-2/anthropic.claude-3-5-sonnet-20240620-v1:0converse/anthropic.claude-3-5-sonnet-20240620-v1:0Test Coverage
Added comprehensive tests covering:
us.,eu.,apac.,jp.,au.,us-gov.,global.llama/,deepseek_r1/)converse/,invoke/)Related Issues