Ability to pass error messages from API to UI

[nenb]

Feature description

The ability to pass any error message that is raised in the API to the UI. Currently this is not possible as StreamingResponse is used which results in peer closed connection without sending complete message body being raised in the UI for all API errors.

Value and/or benefit

Better user experience eg ability to determine more easily whether an issue is UI or API-related.

Anything else?

No response

[pmeier]

All for it. The message (or chunks in case of streaming) that we are returning always have a role attached to them:

ragna/ragna/deploy/_api/schemas.py

Lines 54 to 57 in 3cef0f7

	class Message(BaseModel):
	id: uuid.UUID = Field(default_factory=uuid.uuid4)
	content: str
	role: ragna.core.MessageRole

Currently MessageRole.ASSISTANT is hardcoded for this

ragna/ragna/core/_rag.py

Lines 223 to 227 in 3cef0f7

	answer = Message(
	content=self._run_gen(self.assistant.answer, prompt, sources),
	role=MessageRole.ASSISTANT,
	sources=sources,
	)

We could switch the role to MessageRole.SYSTEM in case we hit an error. With that we could send the error message either instead of the answer or as last chunk when streaming. If we do that, we could make it protocol, i.e. document, that if one receives a system message when one from the assistant is expected, this indicates an error and it should be displayed as such.

Thoughts?

[nenb]

If we do that, we could make it protocol, i.e. document, that if one receives a system message when one from the assistant is expected, this indicates an error and it should be displayed as such.

It seems a little bit complicated/non-standard, but I would be okay with it.

In practice, whenever I hit an error, it always comes from trying to generate the first chunk. If the first chunk is successful, then I don't remember hitting an error (although it's of course possible).

This was the motivation for #410 - test the first chunk, and if there is an error then return a standard 500 response with details. I suspect that this will catch the vast majority of errors, and is what seemed to be recommended by others when I did a quick web-search.

It might be useful for others who only consume the web API (ie don't use our UI) to have the ability to error before starting streaming.

Happy for you to make a decision on how to proceed.