Uvicorn >=0.30 dies on Kubernetes only #2450
Replies: 8 comments 11 replies
-
|
Similar problem here, processes works fine for a limited amount of time, then dies and restarts with no given reason. These restarts lead to front nginx throws 502 Bad Gateway. Log of uvicorn follows: Even |
Beta Was this translation helpful? Give feedback.
-
|
Someone suggested adding tini as a wrapper / main process; I can confirm this does not make any difference (on my setup) |
Beta Was this translation helpful? Give feedback.
-
|
Another 50 cents: low memory could cause that, but I can't confirm it. |
Beta Was this translation helpful? Give feedback.
-
|
One of the interfaces of my fastapi is designed to handle PPT files and uses a process pool. I have observed that when it processes very large PPT files (~30Mb), the child processes exit and then restart for no apparent reason: It doesn't show any errors, and I can't see errors using trace level logging. |
Beta Was this translation helpful? Give feedback.
-
|
docker image: python:3.10.14-bookworm |
Beta Was this translation helpful? Give feedback.
-
|
Hey, I've been checking this for some hours. I'm able to reproduce it in low/limited resource environments. For example, limiting a docker CPU to 0.2 with 2 workers shows some "Child process [x] died" messages. It happens since version 0.30.0. Reproducible example
from fastapi import FastAPI
# import time
def app():
api = FastAPI()
return api
if __name__ == "__main__":
import uvicorn
uvicorn.run("main:app", host="0.0.0.0", port=8000, factory=True, workers=2)
Build and run with low resources: docker build -t app . && docker run -p 8000:8000 --cpus=".2" appExpected output (I get 1 "Child process [x] died"): If we test this example with uvicorn==0.29.0 it starts up normally: Getting more Child process died messages If we test this example with uvicorn==0.29.0 it starts up normally: Not getting the error with slightly different code, example 1 Move the import inside app function like so: I get a clean startup: Not getting the error with slightly different code, example 2 Changle Dockerfile to run uvicorn directly: Use the minimum FastAPI factory with the import outside: from fastapi import FastAPI
def app():
api = FastAPI()
return apiI get a clean startup: Would you mind @Kludex or @abersheeran opening an issue based on these evidences? Or if this is something expected with the new multiprocess manager implementation released in 0.30.0 at least clarify it? Thanks a lot! |
Beta Was this translation helpful? Give feedback.
-
|
downgrade uvicorn to version0.29.0. It works for me. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I am personally running unicorn directly and still getting the issue, but it is a complex app with many imports
Odd that perhaps it's gotten more resource intensive, I'll see if I can get different behaviour with different CPU limits, thank you for the research
|
Beta Was this translation helpful? Give feedback.
-
|
As also mentioned below, I have attempted removing all resource limits, to no avail. I have noticed that when running the docker image locally it appears to be much more CPU intensive when running under uvicorn >= 0.30 than 0.29 and all our k8s worker nodes are heavily CPU overprovisioned, so this may be the root cause. |
Beta Was this translation helpful? Give feedback.
-
|
Dropping the worker count drastically does allow it to run again normally |
Beta Was this translation helpful? Give feedback.
-
|
#2397 Could you please try out this PR? It might solve the problem. If it is confirmed to be a solution, we will expedite the merging of this PR. |
Beta Was this translation helpful? Give feedback.
-
|
I have tried now your PR with a higher value than the default with the example that I sent and it worked. What I don't fully understand is why my example 2 was working compared to the first one, because at the end it is loading the same amount of imports right? |
Beta Was this translation helpful? Give feedback.
-
|
For me the PR doesn't change anything, even setting the timeout up to 120 seconds. It will eventually have enough living workers to start accepting requests, but the logs are still 90% "Child process died". As I'll mention above really quick, I have attempted removing all resource limits, to no avail. I have noticed that when running the docker image locally it appears to be much more CPU intensive when running under uvicorn >= 0.30 than 0.29 and all our k8s worker nodes are heavily CPU overprovisioned, so this may be the root cause. |
Beta Was this translation helpful? Give feedback.
-
|
is there any workaround for this? im facing a similar issue using google cloud run |
Beta Was this translation helpful? Give feedback.
-
|
My guess is that the program you are running requires more memory than the CloudRun configuration allows. We also have Uvicorn running on the CloudRun platform, and for such an auto-scaling platform, there is no need to specify multiple workers, which would increase memory usage in a single container. |
Beta Was this translation helpful? Give feedback.
-
|
same here. I encountered issues running my FastAPI project using different versions of Uvicorn within a Docker container. Details are as follows:
FROM python:3.12-alpine
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories \
&& apk add --update caddy gcc musl-dev libffi-dev
WORKDIR /app
COPY Backend/requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt -i https://mirrors.aliyun.com/pypi/simple
COPY Backend ./backend
EXPOSE 8000
COPY startup.sh /app/startup.sh
RUN chmod +x /app/startup.sh
CMD ["/app/startup.sh"] My #!/bin/sh
set -e
cd /app/backend
exec uvicorn main:get_app \
--host 0.0.0.0 \
--port 8000 \
--workers 2 \
--proxy-headers \
--forwarded-allow-ips '*' \
--factory \
--log-config logging_config.yaml
uvicorn main:get_app --host 0.0.0.0 --port 8000 --workers 2 --proxy-headers --factory --log-config logging_config.yaml |
Beta Was this translation helpful? Give feedback.
-
|
New Findings: |
Beta Was this translation helpful? Give feedback.
-
|
🤔 Python 3.12.21 what‘s change? |
Beta Was this translation helpful? Give feedback.
-
|
wow after reading this thread and reverting to 0.29 it completely fixed a weird, somewhat transient long running problem I've had where sometimes but not always the app would take 2-4 minutes to start up |
Beta Was this translation helpful? Give feedback.
-
I have a FastAPI app that runs via uvicorn, in a Docker container on Kubernetes. Recently I attempted to upgrade uvicorn to the latest version, but when upgrading past 0.29 (into 0.30 and above), the webserver never comes alive. All worker processes seem to just die (log at the bottom). The odd part is that this does not happen at all locally, even when using the exact same Docker image.
I found some references to a similar issue, which notes it should be fixed in 0.30.2, but in my experience 0.29 is the most recent version that works.
Does anyone have a similar experience? If not, I might try to build a repro example, if the issue is not too specific to just our cluster...
Log (repeats at infinitum untill killed by kubernetes):
Beta Was this translation helpful? Give feedback.
All reactions