FastAPI root_path "/" killed my haystack rest_api Docker deployments

[TobiasOtto]

Hi @ZanSara,
I am reaching out, because something broke my Haystack REST API deployments, when I baked the Docker images myself from scratch. (And I actually had to invest over a day to figure it out)

In short: Suddenly the REST API responded "Not Found" to each and every request. When digging deeper, I noticed, that setting root_path="/" for FastAPI requires a double slash to access all the routes.

Here are examples (notice the double slashes in the last curl command):

ROOT_PATH=""
curl localhost:8100/hs_version
{"hs_version":"1.23.0"}

ROOT_PATH="/"
curl localhost:8100/hs_version
{"detail":"Not Found"}
curl localhost:8100//hs_version
{"hs_version":"1.23.0"}

Maybe it helps? Maybe I am doing something wrong? (Could be a change in FastAPI's behaviour?)
I noticed a few discussions about root_path and (reverse) proxy setups. But the thing I stumbled on is, that it "suddenly" stopped working, when baking the Docker images myself.

Best,
Tobias.

[ZanSara]

Hello! This seems definitely to be a change in FastAPI, specifically in Starlette. I might have found even the PR that introduces this change: encode/starlette#2400

I'm surprised it wasn't highlighted more on their end, since it seems to be a breaking change. It might be worth letting them know about your setup and the fact that the recent release broke it.

[TuanaCelik]

Hey @ZanSara and @TobiasOtto - a user on Discord also experienced something similar with FastAPI and found a solution as far as I know. I will ask them whether they can check out this discussion.

[lambda-science]

Hi @ZanSara, I am reaching out, because something broke my Haystack REST API deployments, when I baked the Docker images myself from scratch. (And I actually had to invest over a day to figure it out)

In short: Suddenly the REST API responded "Not Found" to each and every request. When digging deeper, I noticed, that setting root_path="/" for FastAPI requires a double slash to access all the routes.

Here are examples (notice the double slashes in the last curl command):

ROOT_PATH=""
curl localhost:8100/hs_version
{"hs_version":"1.23.0"}

ROOT_PATH="/"
curl localhost:8100/hs_version
{"detail":"Not Found"}
curl localhost:8100//hs_version
{"hs_version":"1.23.0"}

Maybe it helps? Maybe I am doing something wrong? (Could be a change in FastAPI's behaviour?) I noticed a few discussions about root_path and (reverse) proxy setups. But the thing I stumbled on is, that it "suddenly" stopped working, when baking the Docker images myself.

Best, Tobias.

Hey Tobias,
Well, first of all, you're not alone, the lastest update also killed my whole FastAPI setup that only returned 404 for every routes. Can you please tell me what version of haystack you're using ?
For me everything worked fine with v2.0.0b4 but v2.0.0b5 killed it all. Maybe you're even on v1.x ?
I managed to fix it but honestly I have no real idea how... because it's a silent error and crash. The last thing I did and worked was modify the rest_api (FastAPI code) to remove global variables app and pipelines.

But if you're using pre-baked Docker Image, then no it's not possible to do I guess.
If you're using Haystack's pre-built rest-api, maybe you could set the ROOT_PATH env variable from "/" to "" as a temporary fix and see what happens ? (this environnement variable is natively supported so it could do the job ?)

[TuanaCelik]

Update @TobiasOtto

Seems that there was a breaking change in fast API, additionally, my bad, but seems you and @lambda-science are on different Haystack versions so you are actually talking about the Haystack Rest API that comes with 1.x..

For 1.x, we're pinning the fast API version now for rest_api and you can retry once that's done. Hopefully this will fix your issue..

Edit: It's pinned now 👍

[lambda-science]

But actually I use the old rest_api from haystack 1.x that I ported to be compatible with haystack 2.x so it makes sens that we have the same issue :)

[TobiasOtto]

Thanks for all your answers :-)
Yes, I am on haystack 1.x and for now I just changed my config with ROOT_PATH="". Problem solved (for me).
I was hoping to help with this information to save others some time to figure out, what's going on ;-)