500 Error with Anthropic Claude 3 Opus model on Vertex AI

[whalygood]

I realize that support for the Claude 3 Opus model on Vertex AI Model Garden is still very new, so I took the opportunity to test it out. Unfortunately, I kept hitting this 500 error. A service account with the "Vertex AI User" role is assigned to the GCP Compute instance that this is running on.

{"error":{"message":"VertexAIException - Error code: 500 - {'error': {'code': 500, 'message': 'Internal error encountered.', 'status': 'INTERNAL'}}\nmodel: vertex_ai/claude-3-opus@20240229\n","type":null,"param":null,"code":500}}

Here are the details of the configuration:

docker-compose.yml

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    volumes:
      - ./litellm/config.yaml:/app/config.yaml
    ports:
      - "127.0.0.1:3000:3000"
    command:
      - /bin/sh
      - -c
      - |
        pip install async_generator
        litellm --config '/app/config.yaml' --debug --host 0.0.0.0 --port 3000 --num_workers 8
    entrypoint: []

litellm/config.yaml

model_list:
  - model_name: claude-3-opus
    litellm_params:
      model: vertex_ai/claude-3-opus@20240229
      vertex_project: "project-id-123456"
      vertex_location: "us-central1"
      max_tokens: 4096
litellm_settings: # module level litellm settings - https://github.com/BerriAI/litellm/blob/main/litellm/__init__.py
  drop_params: True
  set_verbose: True

general_settings:
  master_key: sk-1234

Here is the cURL call which I tried:

curl --location http://localhost:3000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{
     "model": "claude-3-opus",
     "messages": [
         {
         "role": "user",
         "content": "what llm are you"
         }
     ],
   }'

Here are the logs of the litellm container:

litellm-1  | Requirement already satisfied: async_generator in /usr/local/lib/python3.11/site-packages (1.10)
litellm-1  | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
litellm-1  | INFO:     Started server process [10]
litellm-1  | INFO:     Waiting for application startup.
litellm-1  | 22:22:45 - LiteLLM Router:INFO: router.py:299 - Intialized router with Routing strategy: simple-shuffle
litellm-1  |
litellm-1  | Routing fallbacks: None
litellm-1  |
litellm-1  | Routing context window fallbacks: None
litellm-1  | INFO:     Application startup complete.
litellm-1  | INFO:     Uvicorn running on http://0.0.0.0:3000 (Press CTRL+C to quit)
litellm-1  | 22:22:55 - LiteLLM:INFO: utils.py:1133 - {'model': 'claude-3-opus@20240229', 'messages': [{'role': 'user', 'content': 'what llm are you'}], 'optional_params': {'max_tokens': 4096}, 'litellm_params': {'acompletion': True, 'api_key': None, 'force_timeout': 600, 'logger_fn': None, 'verbose': False, 'custom_llm_provider': 'vertex_ai', 'api_base': '', 'litellm_call_id': '80404e9c-9aa0-4a6c-9c45-93bec13afcd3', 'model_alias_map': {}, 'completion_call_id': None, 'metadata': {'user_api_key': 'sk-1234', 'user_api_key_alias': None, 'user_api_key_user_id': 'default_user_id', 'user_api_key_org_id': None, 'user_api_key_team_id': None, 'user_api_key_metadata': {}, 'headers': {'host': 'localhost:3000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '150'}, 'endpoint': 'http://localhost:3000/chat/completions', 'model_group': 'claude-3-opus', 'deployment': 'vertex_ai/claude-3-opus@20240229', 'model_info': {'id': 'f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68'}, 'caching_groups': None}, 'model_info': {'id': 'f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68'}, 'proxy_server_request': {'url': 'http://localhost:3000/chat/completions', 'method': 'POST', 'headers': {'host': 'localhost:3000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'authorization': 'Bearer sk-1234', 'content-length': '150'}, 'body': {'model': 'claude-3-opus', 'messages': [{'role': 'user', 'content': 'what llm are you'}]}}, 'preset_cache_key': None, 'no-log': False, 'stream_response': {}}, 'start_time': datetime.datetime(2024, 4, 18, 22, 22, 51, 732436), 'stream': False, 'user': 'default_user_id', 'call_type': 'acompletion', 'litellm_call_id': '80404e9c-9aa0-4a6c-9c45-93bec13afcd3', 'completion_start_time': None, 'max_tokens': 4096, 'vertex_project': 'project-id-123456', 'vertex_location': 'us-central1', 'input': [{'role': 'user', 'content': [{'type': 'text', 'text': 'what llm are you'}]}], 'api_key': None, 'additional_args': {'complete_input_dict': {'max_tokens': 4096}}, 'log_event_type': 'pre_api_call'}
litellm-1  |
litellm-1  |
litellm-1  | #------------------------------------------------------------#
litellm-1  | #                                                            #
litellm-1  | #         'The worst thing about this product is...'          #
litellm-1  | #        https://github.com/BerriAI/litellm/issues/new        #
litellm-1  | #                                                            #
litellm-1  | #------------------------------------------------------------#
litellm-1  |
litellm-1  |  Thank you for using LiteLLM! - Krrish & Ishaan
litellm-1  |
litellm-1  |
litellm-1  |
litellm-1  | Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
litellm-1  |
litellm-1  |
litellm-1  | LiteLLM: Proxy initialized with Config, Set models:
litellm-1  |     claude-3-opus
litellm-1  | LiteLLM Proxy: INITIALIZING LITELLM CALLBACKS!
litellm-1  | callback: <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x792608f20e10>
litellm-1  | callback: <litellm.proxy.hooks.tpm_rpm_limiter._PROXY_MaxTPMRPMLimiter object at 0x79260acb8450>
litellm-1  | callback: <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x792608f20e50>
litellm-1  | callback: <bound method ProxyLogging.response_taking_too_long_callback of <litellm.proxy.utils.ProxyLogging object at 0x792608f20c90>>
litellm-1  | callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x792608fc8710>>
litellm-1  | callback: <litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x792608f20dd0>
litellm-1  | get cache: cache key: 88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b; local_only: False
litellm-1  | get cache: cache result: None
litellm-1  | InMemoryCache: set_cache
litellm-1  | LiteLLM Proxy: Inside Proxy Logging Pre-call hook!
litellm-1  | Inside Max Parallel Request Pre-Call Hook
litellm-1  | get cache: cache key: sk-1234::2024-04-18-22-22::request_count; local_only: False
litellm-1  | get cache: cache result: None
litellm-1  | current: None
litellm-1  | Inside Max TPM/RPM Limiter Pre-Call Hook - token='88dc28d0f030c55ed4ab77ed8faf098196cb1c05df778539800c9f1243fe6b4b' key_name=None key_alias=None spend=0.0 max_budget=None expires=None models=[] aliases={} config={} user_id='default_user_id' team_id=None max_parallel_requests=None metadata={} tpm_limit=None rpm_limit=None budget_duration=None budget_reset_at=None allowed_cache_controls=[] permissions={} model_spend={} model_max_budget={} soft_budget_cooldown=False litellm_budget_table=None org_id=None user_id_rate_limits=None team_id_rate_limits=None team_spend=None team_tpm_limit=None team_rpm_limit=None team_max_budget=None team_models=[] team_blocked=False soft_budget=None team_model_aliases=None api_key='sk-1234' user_role='proxy_admin'
litellm-1  | get cache: cache key: sk-1234; local_only: False
litellm-1  | get cache: cache result: None
litellm-1  | get cache: cache key: default_user_id; local_only: False
litellm-1  | get cache: cache result: None
litellm-1  | _set_limits: False
litellm-1  | Inside Max Budget Limiter Pre-Call Hook
litellm-1  | get cache: cache key: default_user_id_user_api_key_user_id; local_only: False
litellm-1  | get cache: cache result: None
litellm-1  | Inside Cache Control Check Pre-Call Hook
litellm-1  | LiteLLM Proxy: final data being sent to completion call: {'model': 'claude-3-opus', 'messages': [{'role': 'user', 'content': 'what llm are you'}], 'proxy_server_request': {'url': 'http://localhost:3000/chat/completions', 'method': 'POST', 'headers': {'host': 'localhost:3000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'authorization': 'Bearer sk-1234', 'content-length': '150'}, 'body': {'model': 'claude-3-opus', 'messages': [{'role': 'user', 'content': 'what llm are you'}]}}, 'user': 'default_user_id', 'metadata': {'user_api_key': 'sk-1234', 'user_api_key_alias': None, 'user_api_key_user_id': 'default_user_id', 'user_api_key_org_id': None, 'user_api_key_team_id': None, 'user_api_key_metadata': {}, 'headers': {'host': 'localhost:3000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '150'}, 'endpoint': 'http://localhost:3000/chat/completions'}, 'request_timeout': 600}
litellm-1  | get cache: cache key: 22-22:cooldown_models; local_only: False
litellm-1  | get cache: cache result: None
litellm-1  | get cache: cache key: f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68; local_only: True
litellm-1  | get cache: cache result: None
litellm-1  | set cache: key: f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68; value: 1
litellm-1  | InMemoryCache: set_cache
litellm-1  | get cache: cache key: f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68_async_client; local_only: True
litellm-1  | get cache: cache result: None
litellm-1  | get cache: cache key: f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68_async_client; local_only: True
litellm-1  | get cache: cache result: None
litellm-1  | get cache: cache key: f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68_rpm_client; local_only: True
litellm-1  | get cache: cache result: None
litellm-1  |
litellm-1  |
litellm-1  | Request to litellm:
litellm-1  | litellm.acompletion(model='vertex_ai/claude-3-opus@20240229', max_retries=0, vertex_project='project-id-123456', vertex_location='us-central1', max_tokens=4096, messages=[{'role': 'user', 'content': 'what llm are you'}], caching=False, client=None, timeout=6000, proxy_server_request={'url': 'http://localhost:3000/chat/completions', 'method': 'POST', 'headers': {'host': 'localhost:3000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'authorization': 'Bearer sk-1234', 'content-length': '150'}, 'body': {'model': 'claude-3-opus', 'messages': [{'role': 'user', 'content': 'what llm are you'}]}}, user='default_user_id', metadata={'user_api_key': 'sk-1234', 'user_api_key_alias': None, 'user_api_key_user_id': 'default_user_id', 'user_api_key_org_id': None, 'user_api_key_team_id': None, 'user_api_key_metadata': {}, 'headers': {'host': 'localhost:3000', 'user-agent': 'curl/7.81.0', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '150'}, 'endpoint': 'http://localhost:3000/chat/completions', 'model_group': 'claude-3-opus', 'deployment': 'vertex_ai/claude-3-opus@20240229', 'model_info': {'id': 'f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68'}, 'caching_groups': None}, request_timeout=600, model_info={'id': 'f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68'})
litellm-1  |
litellm-1  |
litellm-1  | Initialized litellm callbacks, Async Success Callbacks: [<litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x792608f20dd0>, <litellm.proxy.hooks.tpm_rpm_limiter._PROXY_MaxTPMRPMLimiter object at 0x79260acb8450>, <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x792608f20e10>, <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x792608f20e50>]
litellm-1  | callback: <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x792608f20e10>
litellm-1  | callback: <litellm.proxy.hooks.tpm_rpm_limiter._PROXY_MaxTPMRPMLimiter object at 0x79260acb8450>
litellm-1  | callback: <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x792608f20e50>
litellm-1  | callback: <bound method ProxyLogging.response_taking_too_long_callback of <litellm.proxy.utils.ProxyLogging object at 0x792608f20c90>>
litellm-1  | callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x792608fc8710>>
litellm-1  | callback: <litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x792608f20dd0>
litellm-1  | self.optional_params: {}
litellm-1  | litellm.cache: None
litellm-1  | kwargs[caching]: False; litellm.cache: None
litellm-1  | Final returned optional params: {'max_tokens': 4096, 'vertex_project': 'project-id-123456', 'vertex_location': 'us-central1'}
litellm-1  | self.optional_params: {'max_tokens': 4096, 'vertex_project': 'project-id-123456', 'vertex_location': 'us-central1'}
litellm-1  | _is_function_call: False
litellm-1  | VERTEX AI: vertex_project=project-id-123456; vertex_location=us-central1; vertex_credentials=None
litellm-1  | Logging Details: logger_fn - None | callable(logger_fn) - False
litellm-1  | Logging Details LiteLLM-Failure Call
litellm-1  | get cache: cache key: 22-22:cooldown_models; local_only: False
litellm-1  | get cache: cache result: None
litellm-1  | set cache: key: 22-22:cooldown_models; value: ['f41742da96f2dccd27c815713221da9c8db9c17d580323230c17ce9e736c2b68']
litellm-1  | InMemoryCache: set_cache
litellm-1  | Inside Max Parallel Request Failure Hook
litellm-1  | user_api_key: sk-1234
litellm-1  | get cache: cache key: sk-1234::2024-04-18-22-22::request_count; local_only: False
litellm-1  | get cache: cache result: None
litellm-1  | 22:22:58 - LiteLLM Router:INFO: router.py:518 - litellm.acompletion(model=vertex_ai/claude-3-opus@20240229) Exception VertexAIException - Error code: 500 - {'error': {'code': 500, 'message': 'Internal error encountered.', 'status': 'INTERNAL'}}
litellm-1  | model: vertex_ai/claude-3-opus@20240229
litellm-1  |
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 319, in acompletion
litellm-1  |     response = await init_response
litellm-1  |                ^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/vertex_ai_anthropic.py", line 388, in async_completion
litellm-1  |     message = await vertex_ai_client.messages.create(**data)  # type: ignore
litellm-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/resources/messages.py", line 1474, in create
litellm-1  |     return await self._post(
litellm-1  |            ^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1738, in post
litellm-1  |     return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1441, in request
litellm-1  |     return await self._request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1517, in _request
litellm-1  |     return await self._retry_request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1563, in _retry_request
litellm-1  |     return await self._request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1517, in _request
litellm-1  |     return await self._retry_request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1563, in _retry_request
litellm-1  |     return await self._request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1532, in _request
litellm-1  |     raise self._make_status_error_from_response(err.response) from None
litellm-1  | anthropic.InternalServerError: Error code: 500 - {'error': {'code': 500, 'message': 'Internal error encountered.', 'status': 'INTERNAL'}}
litellm-1  |
litellm-1  | During handling of the above exception, another exception occurred:
litellm-1  |
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 3655, in chat_completion
litellm-1  |     responses = await asyncio.gather(
litellm-1  |                 ^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 413, in acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 409, in acompletion
litellm-1  |     response = await self.async_function_with_fallbacks(**kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 1261, in async_function_with_fallbacks
litellm-1  |     raise original_exception
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 1182, in async_function_with_fallbacks
litellm-1  |     response = await self.async_function_with_retries(*args, **kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 1373, in async_function_with_retries
litellm-1  |     raise original_exception
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 1278, in async_function_with_retries
litellm-1  |     response = await original_function(*args, **kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 523, in _acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 507, in _acompletion
litellm-1  |     response = await _response
litellm-1  |                ^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 3451, in wrapper_async
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 3283, in wrapper_async
litellm-1  |     result = await original_function(*args, **kwargs)
litellm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 340, in acompletion
litellm-1  |     raise exception_type(
litellm-1  |           ^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 8628, in exception_type
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 7877, in exception_type
litellm-1  |     raise APIError(
litellm-1  | litellm.exceptions.APIError: VertexAIException - Error code: 500 - {'error': {'code': 500, 'message': 'Internal error encountered.', 'status': 'INTERNAL'}}
litellm-1  | model: vertex_ai/claude-3-opus@20240229
litellm-1  |
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 319, in acompletion
litellm-1  |     response = await init_response
litellm-1  |                ^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/vertex_ai_anthropic.py", line 388, in async_completion
litellm-1  |     message = await vertex_ai_client.messages.create(**data)  # type: ignore
litellm-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/resources/messages.py", line 1474, in create
litellm-1  |     return await self._post(
litellm-1  |            ^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1738, in post
litellm-1  |     return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1441, in request
litellm-1  |     return await self._request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1517, in _request
litellm-1  |     return await self._retry_request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1563, in _retry_request
litellm-1  |     return await self._request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1517, in _request
litellm-1  |     return await self._retry_request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1563, in _retry_request
litellm-1  |     return await self._request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/anthropic/_base_client.py", line 1532, in _request
litellm-1  |     raise self._make_status_error_from_response(err.response) from None
litellm-1  | anthropic.InternalServerError: Error code: 500 - {'error': {'code': 500, 'message': 'Internal error encountered.', 'status': 'INTERNAL'}}
litellm-1  |
litellm-1  | During handling of the above exception, another exception occurred:
litellm-1  |
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 3655, in chat_completion
litellm-1  |     responses = await asyncio.gather(
litellm-1  |                 ^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 413, in acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 409, in acompletion
litellm-1  |     response = await self.async_function_with_fallbacks(**kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 1261, in async_function_with_fallbacks
litellm-1  |     raise original_exception
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 1182, in async_function_with_fallbacks
litellm-1  |     response = await self.async_function_with_retries(*args, **kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 1373, in async_function_with_retries
litellm-1  |     raise original_exception
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 1278, in async_function_with_retries
litellm-1  |     response = await original_function(*args, **kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 523, in _acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 507, in _acompletion
litellm-1  |     response = await _response
litellm-1  |                ^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 3451, in wrapper_async
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 3283, in wrapper_async
litellm-1  |     result = await original_function(*args, **kwargs)
litellm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 340, in acompletion
litellm-1  |     raise exception_type(
litellm-1  |           ^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 8628, in exception_type
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 7877, in exception_type
litellm-1  |     raise APIError(
litellm-1  | litellm.exceptions.APIError: VertexAIException - Error code: 500 - {'error': {'code': 500, 'message': 'Internal error encountered.', 'status': 'INTERNAL'}}
litellm-1  | model: vertex_ai/claude-3-opus@20240229
litellm-1  |
litellm-1  | updated_value in failure call: {'current_requests': 0, 'current_tpm': 0, 'current_rpm': 0}
litellm-1  | set cache: key: sk-1234::2024-04-18-22-22::request_count; value: {'current_requests': 0, 'current_tpm': 0, 'current_rpm': 0}
litellm-1  | InMemoryCache: set_cache
litellm-1  | INFO:     192.168.80.1:39808 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error

[Manouchehri]

Change your region. Try this:

model_list:
  - model_name: claude-3-opus
    litellm_params:
      model: vertex_ai/claude-3-opus@20240229
      vertex_project: "project-id-123456"
      vertex_location: "us-east5"
      max_tokens: 4096
litellm_settings: # module level litellm settings - https://github.com/BerriAI/litellm/blob/main/litellm/__init__.py
  drop_params: True
  set_verbose: True

general_settings:
  master_key: sk-1234

[whalygood]

Tried switching to us-east5 but ran into a quota error
{"error":{"message":"Error code: 429 - {'error': {'code': 429, 'message': 'Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: anthropic-claude-3-opus. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.', 'status': 'RESOURCE_EXHAUSTED'}}","type":null,"param":null,"code":500}}

It does look like I have 1800 QPS (queries per minute) for "The global QPS of Online Predict on Vertex AI Vision per minute" and same for us-east5 specifically, which should surely be sufficient for a simple cURL test? :) Do you happen to know if enabling the API for the Anthropic model must be requested for each region separately?

[Manouchehri]

Please run this as a test, without LiteLLM:

curl -X POST "https://us-east5-aiplatform.googleapis.com/v1/projects/project-id-123456/locations/us-east5/publishers/anthropic/models/claude-3-opus@20240229:streamRawPredict" \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H 'Content-Type: application/json' \
    -d '{
  "anthropic_version": "vertex-2023-10-16",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Tell me a joke."
        }
      ]
    }
  ],
  "max_tokens": 256,
  "stream": false
}' -v

[whalygood]

Yeah, thanks I think you might be onto something here. Just checked the docs again & found out that the Opus model is only available in us-east5 during the preview. Still getting the 429 error though, despite being opted into the Claude 3 Opus model preview (I filled the form about my use case). Perhaps I might try reaching out to Google support, unless you have any other ideas?

[{
  "error": {
    "code": 429,
    "message": "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: anthropic-claude-3-opus. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.",
    "status": "RESOURCE_EXHAUSTED"
  }
}

[Manouchehri]

Did you accept Opus in the model garden yet? If you did, it should look like this:

[whalygood]

Yes, mine looks exactly the same as in the screenshot.

[achhabra2]

Verify your quota as per this screen shot on the quotas page. Make sure you are looking at the quota assigned specifically to Claude 3 Opus.

If you don't have enough QPM you will see it there. You can request a quota increase through the console.