[Feat]: gpu inference

[kevkid]

Description
Will we see gpu inference to speed up generation

Use Case
In all use cases we want more speed.

[a-ghorbani]

Which device are you referring to? If iPhone, Metal is already supported.

[kevkid]

Android, if I am remembering correctly i was able to compile llamacpp on my device and it ran fairly quick. But even the 3b model feels very slow

[JasonOSX]

Also interesting, I downloaded Qwen-2.5-3B and it worked great, then downloaded some more models and all a sudden all models are extremely slow, only producing 0.5 token per second. it is 6 t/s when removing and installing again. Pixel 8 / Android 15

[sotwi]

Android GPU support would be very welcomed