'ramalama ps' returns exception on macOS when no container-based llms are running #488

planetf1 · 2024-11-24T16:41:59Z

On macOS Sonama I have one model running. This appears not to be running as a container but is working well, with Apple Silicon metal/gpu support. podman ps shows a variety of other containers (of mine) running, but no llms from ralalama

The command should not hit an exception - either it should report that no containers are running, or -- better -- would be to include any non-container llms being served. In my case that is

  501 27676  1627   0  4:35pm ttys021    0:00.80 llama-server --port 9999 -m /Users/jonesn/.local/share/ramalama/models/ollama/llama3.1:8b --host 0.0.0.0

Here's the failing command:

~ ramalama ps
Traceback (most recent call last):
  File "/opt/homebrew/bin/ramalama", line 92, in <module>
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/opt/homebrew/bin/ramalama", line 67, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/opt/homebrew/share/ramalama/ramalama/cli.py", line 390, in list_containers
    if len(_list_containers(args)) == 0:
           ~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/homebrew/share/ramalama/ramalama/cli.py", line 380, in _list_containers
    output = run_cmd(conman_args).stdout.decode("utf-8").strip()
             ~~~~~~~^^^^^^^^^^^^^
  File "/opt/homebrew/share/ramalama/ramalama/common.py", line 96, in run_cmd
    return subprocess.run(args, check=True, cwd=cwd, stdout=stdout, stderr=stderr)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py", line 554, in run
    with Popen(*popenargs, **kwargs) as process:
         ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py", line 1036, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        pass_fds, cwd, env,
                        ^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
                        gid, gids, uid, umask,
                        ^^^^^^^^^^^^^^^^^^^^^^
                        start_new_session, process_group)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py", line 1837, in _execute_child
    and os.path.dirname(executable)
        ~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "<frozen posixpath>", line 177, in dirname
TypeError: expected str, bytes or os.PathLike object, not NoneType

The text was updated successfully, but these errors were encountered:

ericcurtin · 2024-11-24T17:32:00Z

Sounds like a good idea to me! I guess we just grep the pids in python and display that. Wanna take a go at implementing this @planetf1 ? Thanks for the feedback!

ericcurtin · 2024-11-24T18:30:31Z

And we should try and make this technique portable so that it works across macOS and Linux (whilst adding no dependancies). If we must we can do:

if macos:
do this
else if linux and not using containers:
do this

rhatdan · 2024-11-25T13:32:36Z

Did you have neither docker podman installed?

planetf1 · 2025-01-07T14:59:24Z

I'll recheck as it's been a while. I do have 'podman' installed.

planetf1 · 2025-01-07T15:05:06Z

Just rechecked with the latest ramalama code, and get the same result.

Both podman ps and docker ps return a list of currently running containers (though none relate to ramalama)

planetf1 · 2025-01-07T15:08:22Z

% podman ps -a --filter label=RAMALAMA
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

planetf1 · 2025-01-07T15:16:13Z

If I explicitly set the container engine it works ie:

➜  ramalama git:(main) export RAMALAMA_CONTAINER_ENGINE=podman
➜  ramalama git:(main) ramalama ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

This even works if set to docker (since there is an alias now setup - actually a filesystem link docker->podman)

ramalama git:(main) export RAMALAMA_CONTAINER_ENGINE=docker
➜  ramalama git:(main) ramalama ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

It's only when not set this fails -- so the issue must be in the detection logic. ie.

➜  ramalama git:(main) ramalama ps
Traceback (most recent call last):
  File "/opt/homebrew/bin/ramalama", line 93, in <module>
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/opt/homebrew/bin/ramalama", line 72, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/Users/jonesn/AI/ramalama/ramalama/cli.py", line 407, in list_containers
    if len(_list_containers(args)) == 0:
           ~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/jonesn/AI/ramalama/ramalama/cli.py", line 397, in _list_containers
    output = run_cmd(conman_args).stdout.decode("utf-8").strip()
             ~~~~~~~^^^^^^^^^^^^^
  File "/Users/jonesn/AI/ramalama/ramalama/common.py", line 87, in run_cmd
    return subprocess.run(args, check=True, cwd=cwd, stdout=stdout, stderr=stderr)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py", line 554, in run
    with Popen(*popenargs, **kwargs) as process:
         ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py", line 1036, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        pass_fds, cwd, env,
                        ^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
                        gid, gids, uid, umask,
                        ^^^^^^^^^^^^^^^^^^^^^^
                        start_new_session, process_group)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py", line 1837, in _execute_child
    and os.path.dirname(executable)
        ~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "<frozen posixpath>", line 178, in dirname
TypeError: expected str, bytes or os.PathLike object, not NoneType

planetf1 · 2025-01-07T15:21:12Z

So looking at detection, this line seems to be problematic

since

➜  ramalama git:(main) podman machine list --format '{{ .VMType }}'
applehv

I'm not familar with krunkit - and am using podman desktop 1.15. That being said podman was originally installed (and still is) via homebrew, so there may be some discrepancy here? (though all seems to work)

ericcurtin · 2025-01-07T15:25:01Z

So looking at detection, this line seems to be problematic

since
➜  ramalama git:(main) podman machine list --format '{{ .VMType }}'
applehv
I'm not familar with krunkit - and am using podman desktop 1.15. That being said podman was originally installed (and still is) via homebrew, so there may be some discrepancy here? (though all seems to work)

krunkit is pretty new, it's required for accelerated AI workloads within podman machined.

ericcurtin · 2025-01-07T15:26:25Z

If krunkit doesn't exist we should probably just fall back to no containers.

Although if you want to do all the cool containers stuff on macOS I highly recommend krunkit.

A simple check for "/opt/podman/bin/krunkit" might be enough

planetf1 · 2025-01-07T15:28:22Z

Just reading about it. Optional not default?

containers/podman#22580 (reply in thread)

So ideally ramalama would detect the applehv configuration and advise krunkit is used?

ericcurtin · 2025-01-07T15:39:13Z

Or we could simplify the check to effectively:

podman machine list | grep "Currently running"

ramalama will work within applehv, it just won't be accelerated

planetf1 · 2025-01-07T16:04:32Z

I can confirm that with a krunkit vm ramalama works without further configuration.

In my case the confusion arose from having installed podman via 'homebrew' prior to podman desktop. So many macOS users make use of homebrew, so I imagine that's quite typical. This distribution doesn't seem to include krunkit (not sure if that may change, or if versions specific, though mine was up to date)

Secondly, krunkit isn't the default, even when using podman desktop - though it is easy to select.

I would suggest that there's several options

minimum would just be to document this clearly
appropriate imo would be to allow ramalama to work as long as podman/docker found
even better would be to add a warning/message if you find the vm isn't accelerated

ericcurtin · 2025-01-07T16:21:58Z

I can confirm that with a krunkit vm ramalama works without further configuration.

In my case the confusion arose from having installed podman via 'homebrew' prior to podman desktop. So many macOS users make use of homebrew, so I imagine that's quite typical. This distribution doesn't seem to include krunkit (not sure if that may change, or if versions specific, though mine was up to date)

Secondly, krunkit isn't the default, even when using podman desktop - though it is easy to select.

I would suggest that there's several options

minimum would just be to document this clearly

appropriate imo would be to allow ramalama to work as long as podman/docker found

It's not a sufficient check because if podman machine isn't started:

$ podman ps
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: read tcp 127.0.0.1:55279->127.0.0.1:51791: read: connection reset by peer

even better would be to add a warning/message if you find the vm isn't accelerated

ericcurtin · 2025-01-07T16:22:25Z

Open to PRs in general to make this better!

ericcurtin · 2025-01-09T12:31:05Z

This should fix part of the issue:

#557

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'ramalama ps' returns exception on macOS when no container-based llms are running #488

'ramalama ps' returns exception on macOS when no container-based llms are running #488

planetf1 commented Nov 24, 2024

ericcurtin commented Nov 24, 2024

ericcurtin commented Nov 24, 2024

rhatdan commented Nov 25, 2024

planetf1 commented Jan 7, 2025

planetf1 commented Jan 7, 2025

planetf1 commented Jan 7, 2025

planetf1 commented Jan 7, 2025

planetf1 commented Jan 7, 2025

ericcurtin commented Jan 7, 2025

ericcurtin commented Jan 7, 2025

planetf1 commented Jan 7, 2025

ericcurtin commented Jan 7, 2025

planetf1 commented Jan 7, 2025

ericcurtin commented Jan 7, 2025 •

edited

Loading

ericcurtin commented Jan 7, 2025

ericcurtin commented Jan 9, 2025

'ramalama ps' returns exception on macOS when no container-based llms are running #488

'ramalama ps' returns exception on macOS when no container-based llms are running #488

Comments

planetf1 commented Nov 24, 2024

ericcurtin commented Nov 24, 2024

ericcurtin commented Nov 24, 2024

rhatdan commented Nov 25, 2024

planetf1 commented Jan 7, 2025

planetf1 commented Jan 7, 2025

planetf1 commented Jan 7, 2025

planetf1 commented Jan 7, 2025

planetf1 commented Jan 7, 2025

ericcurtin commented Jan 7, 2025

ericcurtin commented Jan 7, 2025

planetf1 commented Jan 7, 2025

ericcurtin commented Jan 7, 2025

planetf1 commented Jan 7, 2025

ericcurtin commented Jan 7, 2025 • edited Loading

ericcurtin commented Jan 7, 2025

ericcurtin commented Jan 9, 2025

ericcurtin commented Jan 7, 2025 •

edited

Loading