Out of the blue , granian crash without reason on any load/usage . Anyway to debug that?

[v3ss0n]

[ERROR] Unexpected exit from worker-1
[INFO] Shutting down granian

Granian process stopped.
Traceback (most recent call last):
  File "/home/ubuntu/text2sql/.venv/bin/text2sql", line 8, in <module>
    sys.exit(run_cli())
             ^^^^^^^^^
  File "/home/ubuntu/text2sql/src/text2sql/__main__.py", line 26, in run_cli
    run_litestar_cli()
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/litestar/__main__.py", line 6, in run_cli
    litestar_group()
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)                                                                                                                                                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)                                                                                                                                                                                                  ^^^^^^^^^^^^^^^^
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/litestar/cli/_utils.py", line 254, in wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/litestar_granian/cli.py", line 384, in run_command
    _run_granian_in_subprocess(
  File "/home/ubuntu/text2sql/.venv/lib/python3.12/site-packages/litestar_granian/cli.py", line 746, in _run_granian_in_subprocess
    subprocess.run(
  File "/home/ubuntu/.local/share/pdm/python/[email protected]/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/ubuntu/text2sql/.venv/bin/python', '-m', 'granian', 'text2sql.asgi:text2sql', '--no-reload', '--host=0.0.0.0', '--port=8888', '--interface=asgi', '--http=auto', '
--threads=1', '--workers=1', '--threading-mode=workers', '--blocking-threads=1', '--loop=auto', '--no-respawn-failed-workers', '--respawn-interval=3.5', '--backlog=1024', '--http1-keep-alive', '--http1-buffer-s
ize=417792', '--no-http1-pipeline-flush']' returned non-zero exit status 1.

[gi0baro]

Setting the environment variable RUST_BACKTRACE=full should report the effective reason for the crash.

Also, it might worth checking wether using --task-impl=asyncio produces the same issue or not.

[v3ss0n]

thanks , i will try and see if it crash again and report to you.

[vk496]

Same error here too. In the logs I just randomly get:

2025-01-29 10:31:51.530406+00:00[ERROR] Unexpected exit from worker-1
2025-01-29 10:31:51.530758+00:00[INFO] Shutting down granian

PS: I start Granian from python (should I not for some reason?):

import os

import granian
import granian.asgi
from granian.constants import Interfaces

if __name__ == "__main__":

    # Debug on crash?
    os.environ["RUST_BACKTRACE"] = "full"

    granian.Granian(
        "myapp.main:app", address="0.0.0.0", port=8000, interface=Interfaces.ASGI, threads=5, log_access=True
    ).serve()

PS2: RUST_BACKTRACE from there didn't output any backtrace. Perhaps I need to do it before invoking python?

[gi0baro]

Feels like a segfault. Could you test also with task_impl set to asyncuo?

[vk496]

Feels like a segfault. Could you test also with task_impl set to asyncio?

I'm pretty sure this is how is already since I starting it from Python and because of this:

granian/granian/server.py

Line 732 in 22e3145

self.task_impl = TaskImpl.asyncio if anyio is not None else TaskImpl.rust

PS: It's not easy to reproduce, since it's happening on production with a lot of async load

[gi0baro]

I see. Any additional context on the workload to add? Eg: the type of traffic you're handling, websockets Vs http, type of io load in the app, etc

[gi0baro]

Also, this might affect performance, but does it make any difference running on asyncio event loop instead of uvloop?

[v3ss0n]

We are not using websocket but using SSE/EventSource. and the workload is very mininaml we are just testing with 3 developers.