Add Websocket connection

[giovanni-vaccarino]

What feature do you want to see added?

Description

Currently, the chatbot uses a classic REST endpoint to get replies. To improve responsiveness, I propose having also a WebSocket-based connection to support token-by-token streaming. This would enable real-time output rendering in the frontend, where in our context of local running LLM would be a great enhancement in user experience.

Upstream changes

No response

Are you interested in contributing this feature?

No response

[berviantoleo]

Anyway, we may need the current REST endpoint for the fallback option (especially when the user can't make a WebSocket connection).

[kartikeyg0104]

@giovanni-vaccarino Sir I am really interested in this, can you please assign me this issue?

[berviantoleo]

Don't need to be assigned for working on this. If the solution is quite straight forward, you can open the PR.

[giovanni-vaccarino]

@kartikeyg0104 As pointed out by @berviantoleo, you are free to contribute

[kartikeyg0104]

Sir, I have raised the pull request(#47), can you plz once review it.