Ramalama serve cannot be ctrl-c

[vpavlin]

ramalama serve without -d will attach to the TTY, but will not react to CTRL+C

[ericcurtin]

Seems like it's this issue:

containers/podman#14707

[ericcurtin]

Doesn't occur without containers

[vpavlin]

I am not using podman (at the moment):)

(.venv) root@pi5:/home/vpavlin/ramalama# docker version
Client:
 Version:           20.10.24+dfsg1
 API version:       1.41
 Go version:        go1.19.8
 Git commit:        297e128
 Built:             Sat Oct 12 15:19:49 2024
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.24+dfsg1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       5d6db84
  Built:            Sat Oct 12 15:19:49 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.20~ds1
  GitCommit:        1.6.20~ds1-1+b1
 runc:
  Version:          1.1.5+ds1
  GitCommit:        1.1.5+ds1-1+deb12u1
 docker-init:
  Version:          0.19.0
  GitCommit:        

[ericcurtin]

podman or docker (I said containers, just referenced the podman issue as an example), doesn't happen with no containers

[rhatdan]

What does ramalama info show?

[rhatdan]

Looks like you are running in a docker container, which should forward the SIGTERM to the PID 1 of the container, llama-server. If this is not exiting, then I would guess llama-serve is ignoring SIGTERM.

[rhatdan]

Doing some testing it looks like llama-serve is ignoreing SIGINT and SIGTERM

[ericcurtin]

It's supposed to call this in llama.cpp

inline void signal_handler(int signal) {
    if (is_terminating.test_and_set()) {
        // in case it hangs, we can force terminate the server by hitting Ctrl+C twice
        // this is for better developer experience, we can remove when the server is stable enough
        fprintf(stderr, "Received second interrupt, terminating immediately.\n");
        exit(1);
    }

    shutdown_handler(signal);
}

[ericcurtin]

@rhatdan tested sending signals to llama-server with no containers with:

kill 2 PID

and

kill 15 PID

manually. Seemed to work fine.

[ericcurtin]

I can confirm it's reproducible with podman on macOS, could be a podman remote thing though.

[rhatdan]

I reproduced locally and entered the container and the llama-run would not exit with kill TERM or kill INT. Do not know what the difference is.

[vpavlin]

What does ramalama info show?

Well, not much:D

# ramalama --debug info
run_cmd:  docker info --format json
Working directory: None
Ignore stderr: False
Ignore all: False
Command finished with return code: 0
Error: Expecting value: line 1 column 1 (char 0)

[ericcurtin]

It looks like it's both an issue for podman and docker. Probably the best way to fix is in llama.cpp upstream, we should log this there.

[rhatdan]

The interesting thing is what could cause it to work correctly outside of a container but blow up inside of a container. this means it must be examining something about it's environment and blocking the receiving of these signals.