Python gradio_client: when submitting asynchronous jobs too rapidly to a batch server, some never report completed

[danbgoldman]

Describe the bug

This is reproducible, but the exact conditions are a bit flaky:

When submitting many asynchronous jobs very quickly using client.submit, the python client gets stuck, and after a certain point jobs are not actually submitted. Some which are submitted never return "done" even after the server has processed them.

On my laptop, the repro below shows 0 to 20 jobs completed, indefinitely, even though 100 jobs were submitted. The client shows about 40 jobs were passed onto the echo function. The numbers vary with each run and depending on system load, so it's most likely a race condition of some kind.

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Reproduction

server.py:

import gradio as gr

item_count = 0

def echo(text):
    global item_count
    batch_size = len(text)
    item_count += batch_size
    print(f"Batch size: {batch_size} | Total items: {item_count}")
    return [text]

io = gr.Interface(fn=echo, inputs="text", outputs="text", batch=True, max_batch_size=16)
io.queue().launch()

loadtest.py:

import time
from gradio_client import Client, handle_file

CLIENT_HOST = "http://localhost:7860/"

# Submitting jobs too quickly results in some kind of race condition on the client
# side that causes the requests to never complete. On my laptop this happens within
# about 20-30 rapid submissions.
COUNT = 100

client = Client(CLIENT_HOST)
jobs = []
for i in range(COUNT):
  jobs.append(client.submit(text = str(i),api_name="/predict"))

# Wait for all jobs to complete
done = False
while not done:
  time.sleep(0.5)
  jobs_completed = sum([1 for job in jobs if job.done()])
  print(f"{jobs_completed} jobs completed")
  done = jobs_completed == COUNT

Screenshot

No response

Logs

[~/Documents/src/gradio_bug] dangoldman% ./server.py
* Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Batch size: 1 | Total items: 1
Batch size: 8 | Total items: 9
Batch size: 6 | Total items: 15
Batch size: 16 | Total items: 31
Batch size: 9 | Total items: 40
...

[~/Documents/src/gradio_bug] dangoldman% ./loadtest.py
Loaded as API: http://localhost:7860/ ✔
0 jobs completed
0 jobs completed
0 jobs completed
0 jobs completed
0 jobs completed
0 jobs completed
0 jobs completed
...

System Info

Gradio Environment Information:
------------------------------
Operating System: Darwin
gradio version: 5.6.0
gradio_client version: 1.4.3

------------------------------------------------
gradio dependencies in your environment:

aiofiles: 23.2.1
anyio: 4.6.2.post1
audioop-lts: 0.2.1
fastapi: 0.115.5
ffmpy: 0.4.0
gradio-client==1.4.3 is not installed.
httpx: 0.27.2
huggingface-hub: 0.26.2
jinja2: 3.1.4
markupsafe: 2.1.5
numpy: 2.1.3
orjson: 3.10.11
packaging: 24.2
pandas: 2.2.3
pillow: 11.0.0
pydantic: 2.9.2
pydub: 0.25.1
python-multipart==0.0.12 is not installed.
pyyaml: 6.0.2
ruff: 0.7.4
safehttpx: 0.1.1
semantic-version: 2.10.0
starlette: 0.41.2
tomlkit==0.12.0 is not installed.
typer: 0.13.0
typing-extensions: 4.12.2
urllib3: 2.2.3
uvicorn: 0.32.0
authlib; extra == 'oauth' is not installed.
itsdangerous; extra == 'oauth' is not installed.


gradio_client dependencies in your environment:

fsspec: 2024.10.0
httpx: 0.27.2
huggingface-hub: 0.26.2
packaging: 24.2
typing-extensions: 4.12.2
websockets: 12.0

Severity

I can work around it