Detect Ubuntu Asahi during container setup

[chrootchad]

#526 is failing to detect and use the ashi image on Ubuntu Asahi - ref:

Thanks @ericcurtin I ran

ramalama --gpu --debug --image quay.io/ramalama/asahi:0.5.4 run ollama://deepseek-r1:7b

and forced the latest asahi image and it worked (including the gpu support I couldn't get working previously with other models when the quay.io/ramalama/ramalama:latest image was being used)

Seems to be an ubuntu asahi quirk, but it looks like this:

ramalama/ramalama/common.py

Line 203 in 3032571

if os.path.exists('/etc/os-release'):

    # ASAHI CASE
    if os.path.exists('/etc/os-release'):
        with open('/etc/os-release', 'r') as file:
            if "asahi" in file.read().lower():
                # Set Env Var and break
                os.environ["ASAHI_VISIBLE_DEVICES"] = "1"
                return

isn't setting ASAHI_VISIBLE_DEVICES=1
because on ubuntu asahi

cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Doesn't contain "asahi" anywhere in the output, this seems to though

hostnamectl | grep asahi
          Kernel: Linux 6.12.0-1004-asahi-arm

Thanks again for your help.

Originally posted by @chrootchad in #616

[ericcurtin]

You're gonna have to help me with a super cheap detection technique for Asahi.

From the above I guess:

cat /proc/cmdline

contains the "asahi" string too? That might be a portable technique we can use for Fedora and Ubuntu for a while.

I don't want to get your hopes up, even when you pull the Asahi container and it works, the GPU inferencing will be slow, the enablement work isn't done for that. You are likely better off with CPU-based inferencing.

Tagging @asahilina just to make her aware that llama.cpp demand is increasing for Asahi, hence it's also increasing for RamaLama as a user of llama.cpp

[chrootchad]

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-6.12.0-1004-asahi-arm root=UUID=
...

✅

[asahilina]

That only works for systems running downstream kernels, not upstream.

If you just want to know "is this an Apple ARM machine?", the "canonical" way to do that is something like grep "apple,arm-platform" /proc/device-tree/compatible. In Python, b"apple,arm-platform" in open("/proc/device-tree/compatible", "rb").read().split(b"\0") (plus OSError handling for non-DT systems, since the file won't exist on those).

[ericcurtin]

Thanks @asahilina will test that and add to the PR

[chrootchad]

~#cat /proc/device-tree/compatible
apple,j274apple,t8103apple,arm-platform

[rhatdan]

A better way might be to set an environment variable within the image to identify the image type. One issue with doing it based on command line, is that someone could launch service on a different container based on the asahi container.

Lets change the containerfiles to set the GPU type as an environment variable.

ENV ASAHI_VISIBLE_DEVICES 1

During the build, then during the run, there is no issue.

[asahilina]

~#cat /proc/device-tree/compatible
apple,j274apple,t8103apple,arm-platform

Note that there are NULs separating the entries (it's a list of back to back NUL-terminated strings).

[ericcurtin]

A better way might be to set an environment variable within the image to identify the image type. One issue with doing it based on command line, is that someone could launch service on a different container based on the asahi container.

Lets change the containerfiles to set the GPU type as an environment variable.

ENV ASAHI_VISIBLE_DEVICES 1

During the build, then during the run, there is no issue.

We can hardcode this in the Asahi container.

But the reason we try to detect is asahi has a special image (it's a different version of mesa, a downstream fork), just like we detect for ROCm, CUDA and pull the right container. So we would still need the detection code.

[rhatdan]

Ok PR was merged for this, please verify the upstream fixes your issue.

[chrootchad]

@ericcurtin

I don't want to get your hopes up, even when you pull the Asahi container and it works, the GPU inferencing will be slow, the enablement work isn't done for that. You are likely better off with CPU-based inferencing.

I guess this was the source?
ggerganov/llama.cpp#10879

Seems to work well enough..
Not sure why the GPU isn’t consuming more of the unified ram available to it while running r1:8b. There’s 16gb installed, ~14.5 of that available to the GPU and htop shows it using ~ 5gb.
On the other hand r1:14b won’t run and crashes with an out of memory error as I would have expected 🤷‍♂️. First world problems.