Can RamaLama support Kubernetes-based inference clustering on a Mac Mini M4?

[hotwa]

I’m really impressed with the work being done on RamaLama and its ability to handle various models for inference. I have been exploring the possibility of leveraging Kubernetes (K8s) for distributed model in my local environment, specifically on a Mac Mini.

Is there currently support within RamaLama to deploy and manage model inference workloads using Kubernetes on a Mac Mini?
If not natively supported, are there any recommendations or best practices for integrating RamaLama with Kubernetes for local development/testing purposes?
Are there any known limitations or considerations when running Kubernetes-based inference clusters on ARM-based hardware (like the M4 chip) using RamaLama?

[ericcurtin]

It can be done, you need to install podman-machine with krunkit that's the first step https://podman.io/ . There's also kubelet generators via:

ramalama serve --generate

Most of the bits are there, just need someone to tie it all together.

[ericcurtin]

Would make a great blog post!

[rhatdan]

ramalama serve --generate kube MODEL

Will generate a k8s deployment for the containerized AI Model.