Android Backend

[AmSmart]

I was trying to get this up and running with MAUI on Android but as far as I can tell, only desktop support is available for the time being. Will the maintainers be open to a PR that merges android CPU support in?

[martindevans]

Android support would be great!

If you're interested in working on it I'd suggest starting out by modifying compile.yml to produce Android binaries. That script is what we use to produce all of the binaries we distribute. Once that's merged we can start looking at what modifications are required for runtime support (if any).

[AmSmart]

Sounds good, I'll start working on this

[DGdev91]

So.... how' the progress on this?

Some time ago i managed to build up a working example on Android (just an experiment to see if it was possible to run an LLM inside Godot Engine, using the c# backend), just by building myself the libraries and loading them using the native lib classes.

The llama.cpp version used for v.0.15 seems like to have a bug wich makes it impossible to compile on android.
The current upstream builds normally, but i can't make it work, not sure if i'm doing something wrong or it's just because the newer versions introduced some breaking changes (i know, i'm supposed to only use that exact version)

[martindevans]

I introduced the changes into the build pipeline before 0.15 (see here) but they're currently disabled, because when I came to do the 0.15 build it failed on Android. I think it was a temporary upstream issue with Android builds, so I'm expecting to be able to enable that for 0.16.

What we don't have yet is any runtime loading of those binaries. There needs to be an item which dynamically detects when we're on Android, checks what CPU it is running on, and loads the right binaries (see here). That's the next piece of work to be done, if you're interested in contributing towards Android support :)

[DGdev91]

I introduced the changes into the build pipeline before 0.15 (see here) but they're currently disabled, because when I came to do the 0.15 build it failed on Android. I think it was a temporary upstream issue with Android builds, so I'm expecting to be able to enable that for 0.16.

What we don't have yet is any runtime loading of those binaries. There needs to be an item which dynamically detects when we're on Android, checks what CPU it is running on, and loads the right binaries (see here). That's the next piece of work to be done, if you're interested in contributing towards Android support :)

Yes, most likely is the same issue i encountered while attempting to compile the android libs for 0.15. i'm able to compile normally now, so i guess it got fixed upstream.

Anyway... before i'm able to help, or at least give that a try, i need a working example. and right now, i'm having some troubles.
On the old version 0.10.0 i didn't had to do much else, i just copied the libs on the output folder for the c# code and those got loaded. i didn't even had to use "NativeLibraryConfig.Instance.WithLibrary("");" (wich now should be different for llama and llava, but whatever....)
I guess that's because of the version not exactly the same, but can easly be something else

[AmSmart]

Hi @martindevans, I had some time on my hands again and decided to continue with this issue. I saw that lots of nice updates have been made to clean up the build pipeline and the loading logic, making what I have to do so much easier.

I have some drafts already but I'm looking at making my PR as complete as possible before opening. I'm at a junction where I need to add some testing/validation that proves that all works as expected but I'm not exactly sure how this repo handles this, or what would be the preferred route.

I'm thinking of adding an example Android app project which has its own test project. Would that be acceptable? If yes, do you have any preferences on how I structure that with the rest of the solution?

[martindevans]

To be honest I don't know a lot about Android development with C#, so I'm pretty much happy to defer to you on this. Adding an Android demo/test project to the solution sounds like it could work well.

Just a note regarding Android - the stage to compile the binaries has been disabled for a while (here). You may want to investigate that, once it's compiling we can include android binaries in future releases, and that will be a good foundation for your other Android bits :)

[AmSmart]

Sounds good. I'll checkout the build issue as well. It built just fine last I tried so I'm hoping tinkering with a few flags should be enough to get it back building.

Thanks for the quick feedback

[Darlanio]

Looking forward to android support for LLamaSharp.

Tested it with 0.19 and got System.PlatformNotSupportedException: 'Operation is not supported on this platform.' on the ModelParams constructor.

[amirvenus]

As it's been a while, I was wondering if there has been any progress on this?

Thanks

[Darlanio]

Still waiting for this - but I should check if there has been any progress... here or in other open source projects. Thanks for the reminder - I have a MAUI project on my TODO-list which requires some LLM in the backend (locally on the mobile phone / pad)