[Bug]: Cannot get past 50 RPS

[vutrung96]

What happened?

I have OpenAI tier 5 usage, which should give me 30,000 RPM = 500 RPS with "gpt-4o-mini". However I struggle get past 50 RPS.

The minimal replication:

from litellm import acompletion

tasks = [acompletion(
    model="gpt-4o-mini",
    messages=[
      {"role": "system", "content": "You're an agent who answers yes or no"},
      {"role": "user", "content": "Is the sky blue?"},
    ],
) for i in range(2000)]

I only get 50 items/second as opposed to ~500 items/second when sending raw HTTP requests.

Relevant log output

 16%|█████████████████████▌                                                                                                                 | 320/2000 [00:09<00:40, 41.49it/s]

Twitter / LinkedIn details

No response

[ishaan-jaff]

hi @vutrung96 looking into this, how do you get the % complete log output ?

[vutrung96]

Hi @ishaan-jaff I was just using tqdm

[CharlieJCJ]

Hi @ishaan-jaff , any updates on this, also facing this issue!

[ishaan-jaff]

hi @vutrung96 @CharlieJCJ do you see the issue on litellm.router too ? https://docs.litellm.ai/docs/routing

It would help me if you could test with litellm router too

[RyanMarten]

Hi @ishaan-jaff
We tracked down the root cause of the issue.

Litellm uses the official OpenAI python client

client: Optional[Union[OpenAI, AsyncOpenAI]] = None,

The official OpenAI client has performance issues with high numbers of concurrent requests due to issues in httpx

• httpx client has very poor performance for concurrent requests compared to aiohttp openai/openai-python#1596

The issues in httpx are due to a number of factors related to anyio vs asyncio

• Improve async performance. encode/httpx#3215

Which are addressed in the open PRs below

• Use asyncio synchronization instead of anyio encode/httpcore#922
• Connection pool optimization: move socket polling from expiry checks to connection usage encode/httpcore#928
• Connection pool optimization: reduce connection maintenance loops complexity encode/httpcore#929
• Add asyncio backend encode/httpcore#930
• Use AnyIO with fast_acquire synchronization  encode/httpcore#953

We saw this when implementing litellm as the backend for our synthetic data engine

• Add LiteLLM+instructor (for structured output) backend for curator bespokelabsai/curator#141

When using our own openai client (with aiohttp instead of httpx) we saturate the highest rate limits (30,000 requests per minute on gpt-4o-mini tier 5). When using litellm, the performance issues cap us well under the highest rate limit (200 queries per second - 12,000 requests per minute).