v1.55.4

What's Changed

• (feat) Add Azure Blob Storage Logging Integration by @ishaan-jaff in #7265
• (feat) Add Bedrock knowledge base pass through endpoints by @ishaan-jaff in #7267
• docs(input.md): document 'extra_headers' param support by @krrishdholakia in #7268
• fix(utils.py): fix openai-like api response format parsing by @krrishdholakia in #7273
• LITELLM: Remove requests library usage by @krrishdholakia in #7235
• Litellm dev 12 17 2024 p2 by @krrishdholakia in #7277
• Litellm dev 12 17 2024 p3 by @krrishdholakia in #7279
• LiteLLM Minor Fixes & Improvements (12/16/2024) - p1 by @krrishdholakia in #7263
• Add Azure Llama 3.3 by @emerzon in #7283
• (feat) proxy Azure Blob Storage - Add support for AZURE_STORAGE_ACCOUNT_KEY Auth by @ishaan-jaff in #7280
• Correct max_tokens on Model DB by @emerzon in #7284
• (fix) unable to pass input_type parameter to Voyage AI embedding mode by @ishaan-jaff in #7276

Full Changelog: v1.55.3...v1.55.4

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.4

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.4

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	277.54264076081057	6.1096008757758025	0.0	1827	0	217.2655560000294	1776.9381030000204
Aggregated	Passed ✅	250.0	277.54264076081057	6.1096008757758025	0.0	1827	0	217.2655560000294	1776.9381030000204

v1.55.3

What's Changed

• LiteLLM Minor Fixes & Improvements (12/13/2024) pt.1 by @krrishdholakia in #7219
• (feat - Router / Proxy ) Allow setting budget limits per LLM deployment by @ishaan-jaff in #7220
• build(deps): bump nanoid from 3.3.7 to 3.3.8 in /ui/litellm-dashboard by @dependabot in #7216
• Litellm add router to base llm testing by @ishaan-jaff in #7202
• fix(main.py): fix retries being multiplied when using openai sdk by @krrishdholakia in #7221
• (proxy) - Auth fix, ensure re-using safe request body for checking model field by @ishaan-jaff in #7222
• (UI fix) - Allow editing Key Metadata by @ishaan-jaff in #7230
• (UI) Fix Usage Tab - Don't make expensive UI queries after SpendLogs crosses 1M Rows by @ishaan-jaff in #7229
• (code quality) Add ruff check to ban print in repo by @ishaan-jaff in #7233
• (UI QA) - stop making expensive UI queries when 1M + spendLogs in DB by @ishaan-jaff in #7234
• Fix vllm import by @ivanvykopal in #7224
• Add new Gemini 2.0 Flash model to Vertex AI. by @Manouchehri in #7193
• Litellm remove circular imports by @krrishdholakia in #7232
• (feat) Add Tag-based budgets on litellm router / proxy by @ishaan-jaff in #7236
• Litellm dev 12 14 2024 p1 by @krrishdholakia in #7231

New Contributors

• @ivanvykopal made their first contribution in #7224

Full Changelog: v1.55.2...v1.55.3

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.3

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	281.1265187306242	6.1657490001280255	0.0033418693767631575	1845	1	119.36488499998177	3755.8482019999815
Aggregated	Passed ✅	250.0	281.1265187306242	6.1657490001280255	0.0033418693767631575	1845	1	119.36488499998177	3755.8482019999815

v1.55.1-stable

Full Changelog: v1.55.1...v1.55.1-stable

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_dec_14-stable

litellm-database image

ghcr.io/berriai/litellm-database:litellm_stable_dec_14-stable

litellm-non-root image

ghcr.io/berriai/litellm-non_root:litellm_stable_dec_14-stable

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	200.0	217.72878062246997	6.2754597178458145	0.0033415653449658223	1878	1	76.6410740000083	1257.3869729999956
Aggregated	Passed ✅	200.0	217.72878062246997	6.2754597178458145	0.0033415653449658223	1878	1	76.6410740000083	1257.3869729999956

v1.55.2

What's Changed

• Litellm dev 12 12 2024 by @krrishdholakia in #7203
• Litellm dev 12 11 2024 v2 by @krrishdholakia in #7215

Full Changelog: v1.55.1...v1.55.2

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	282.51255728779716	6.192691226975396	0.0	1852	0	223.9336790000266	3178.0424589999257
Aggregated	Passed ✅	250.0	282.51255728779716	6.192691226975396	0.0	1852	0	223.9336790000266	3178.0424589999257

v1.55.1

What's Changed

• (feat) add response_time to StandardLoggingPayload - logged on datadog, gcs_bucket, s3_bucket etc by @ishaan-jaff in #7199
• build(deps): bump nanoid from 3.3.7 to 3.3.8 in /ui by @dependabot in #7198
• (Feat) DataDog Logger - Add HOSTNAME and POD_NAME to DataDog logs by @ishaan-jaff in #7189
• (feat) add error_code, error_class, llm_provider to StandardLoggingPayload by @ishaan-jaff in #7200
• (docs) Document StandardLoggingPayload Spec by @ishaan-jaff in #7201
• fix: Support WebP image format and avoid token calculation error by @ishaan-jaff in #7182
• (feat) UI - Disable Usage Tab once SpendLogs is 1M+ Rows by @ishaan-jaff in #7208
• (minor fix proxy) Clarify Proxy Rate limit errors are showing hash of litellm virtual key by @ishaan-jaff in #7210
• (fix) latency fix - revert prompt caching check on litellm router by @ishaan-jaff in #7211

Full Changelog: v1.55.0...v1.55.1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	274.17864765330575	6.170501674094568	0.0	1846	0	212.15181599995958	2203.3609819999356
Aggregated	Passed ✅	250.0	274.17864765330575	6.170501674094568	0.0	1846	0	212.15181599995958	2203.3609819999356

v1.55.0.dev2

What's Changed

• (feat) add response_time to StandardLoggingPayload - logged on datadog, gcs_bucket, s3_bucket etc by @ishaan-jaff in #7199
• build(deps): bump nanoid from 3.3.7 to 3.3.8 in /ui by @dependabot in #7198
• (Feat) DataDog Logger - Add HOSTNAME and POD_NAME to DataDog logs by @ishaan-jaff in #7189
• (feat) add error_code, error_class, llm_provider to StandardLoggingPayload by @ishaan-jaff in #7200
• (docs) Document StandardLoggingPayload Spec by @ishaan-jaff in #7201
• fix: Support WebP image format and avoid token calculation error by @ishaan-jaff in #7182
• (feat) UI - Disable Usage Tab once SpendLogs is 1M+ Rows by @ishaan-jaff in #7208

Full Changelog: v1.55.0...v1.55.0.dev2

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.0.dev2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	210.0	236.69042419128075	6.133942906309277	0.0	1835	0	175.69668400000182	4096.7015589999955
Aggregated	Passed ✅	210.0	236.69042419128075	6.133942906309277	0.0	1835	0	175.69668400000182	4096.7015589999955

v1.55.0.dev1

Full Changelog: v1.55.0...v1.55.0.dev1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.0.dev1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	220.0	243.6248328955295	6.208881714875033	0.0	1857	0	195.87834699996165	1936.7717839999727
Aggregated	Passed ✅	220.0	243.6248328955295	6.208881714875033	0.0	1857	0	195.87834699996165	1936.7717839999727

v1.55.0

What's Changed

• Litellm code qa common config by @krrishdholakia in #7113
• (Refactor) Code Quality improvement - use Common base handler for Cohere by @ishaan-jaff in #7117
• (Refactor) Code Quality improvement - Use Common base handler for clarifai/ by @ishaan-jaff in #7125
• (Refactor) Code Quality improvement - Use Common base handler for cloudflare/ provider by @ishaan-jaff in #7127
• (Refactor) Code Quality improvement - Use Common base handler for Cohere /generate API by @ishaan-jaff in #7122
• (Refactor) Code Quality improvement - Use Common base handler for anthropic_text/ by @ishaan-jaff in #7143
• docs: document code quality by @krrishdholakia in #7149
• (Refactor) Code Quality improvement - stop redefining LiteLLMBase by @ishaan-jaff in #7147
• LiteLLM Common Base LLM Config (pt.2) by @krrishdholakia in #7146
• LiteLLM Common Base LLM Config (pt.3): Move all OAI compatible providers to base llm config by @krrishdholakia in #7148
• refactor(sagemaker/): separate chat + completion routes + make them b… by @krrishdholakia in #7151
• rename llms/OpenAI/ -> llms/openai/ by @ishaan-jaff in #7154
• Code Quality improvement - remove symlink to requirements.txt from within litellm by @ishaan-jaff in #7155
• LiteLLM Common Base LLM Config (pt.4): Move Ollama to Base LLM Config by @krrishdholakia in #7157
• Code Quality Improvement - remove file_apis, fine_tuning_apis from /llms by @ishaan-jaff in #7156
• Revert "LiteLLM Common Base LLM Config (pt.4): Move Ollama to Base LLM Config" by @krrishdholakia in #7160
• Litellm ollama refactor by @krrishdholakia in #7162
• Litellm vllm refactor by @krrishdholakia in #7158
• Litellm merge pr by @krrishdholakia in #7161
• Code Quality Improvement - remove tokenizers/ from /llms by @ishaan-jaff in #7163
• build(deps): bump nanoid from 3.3.7 to 3.3.8 in /docs/my-website by @dependabot in #7159
• (Refactor) Code Quality improvement - remove /prompt_templates/ , base_aws_llm.py from /llms folder by @ishaan-jaff in #7164
• Code Quality Improvement - use vertex_ai/ as folder name for vertexAI by @ishaan-jaff in #7166
• Code Quality Improvement - move aleph_alpha to deprecated_providers by @ishaan-jaff in #7168
• (Refactor) Code Quality improvement - rename text_completion_codestral.py -> codestral/completion/ by @ishaan-jaff in #7172
• (Code Quality) - Add test to enforce all folders in /llms are a litellm provider by @ishaan-jaff in #7175
• fix(get_supported_openai_params.py): cleanup by @krrishdholakia in #7176
• fix(acompletion): support fallbacks on acompletion by @krrishdholakia in #7184

Full Changelog: v1.54.1...v1.55.0

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.0

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	286.19507948581224	5.886697197840291	0.0033409178194326278	1762	1	211.68456200001629	3578.4067740000296
Aggregated	Passed ✅	250.0	286.19507948581224	5.886697197840291	0.0033409178194326278	1762	1	211.68456200001629	3578.4067740000296

v1.54.1

What's Changed

• refactor - use consistent file naming convention AI21/ -> ai21 by @ishaan-jaff in #7090
• refactor - use consistent file naming convention AzureOpenAI/ -> azure by @ishaan-jaff in #7092
• Litellm dev 12 07 2024 by @krrishdholakia in #7086

Full Changelog: v1.54.0...v1.54.1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.54.1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Failed ❌	280.0	340.7890831504466	5.986291177372485	0.0	1788	0	236.28402200000664	4047.592437999981
Aggregated	Failed ❌	280.0	340.7890831504466	5.986291177372485	0.0	1788	0	236.28402200000664	4047.592437999981

v1.54.0

What's Changed

• (feat) Track custom_llm_provider in LiteLLMSpendLogs by @ishaan-jaff in #7081
• Add MLflow to the side bar by @B-Step62 in #7031
• (bug fix) SpendLogs update DB catch all possible DB errors for retrying by @ishaan-jaff in #7082
• (Feat) Add StructuredOutputs support for Fireworks.AI by @ishaan-jaff in #7085
• added deepinfra/Meta-Llama-3.1-405B-Instruct to the Model json by @AliSayyah in #7084
• (feat) Add created_at and updated_at for LiteLLM_UserTable by @ishaan-jaff in #7089

New Contributors

• @AliSayyah made their first contribution in #7084

Full Changelog: v1.53.9...v1.54.0

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.54.0

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	200.0	220.2003271503722	6.29832230581454	0.0	1882	0	179.34225999999853	1827.969679000006
Aggregated	Passed ✅	200.0	220.2003271503722	6.29832230581454	0.0	1882	0	179.34225999999853	1827.969679000006