machine learning: hardware acceleration on apple silicon (metal)

[traktuner]

Hello,
just wanted to ask if it's planned to support ML hardware acceleration on apple silicon gpu's (metal).
This would result in a good performance for the initial ML scan on macs with "max" and/or "ultra" CPUs.
Thank you!

[radh21301]

This would be awesome! Even people that have macbook air or pro as a daily driver laptop would benefit if they had immich running on low capability server. Say a pi or something.

Maybe convert this into a feature request instead of 'Q&A'

[Qhilm]

just to clarify: a MacBook is already insanely faster than a Pi, even without the hardware acceleration. My MacBook is 30-50 times faster than my Synology NAS, even though there is no hardware acceleration.

I still would love to have this feature though, because then I can rerun machine learning on my entire library with a different model in a matter of hours (or maybe even faster?) instead of taking two days 🙂

[radh21301]

updated my comment. I meant - many users have a macbook air or pro as a daily driver laptop. but use a pi as a server for immich. They would definitely befefit like you say.

[bo0tzz]

To my understanding, docker on mac runs inside of a Linux VM and it's not possible to pass the Apple Metal hwaccel through to that.

[xobust]

Seems to be possible using GPU virtio-gpu.
See this post using podman and llama https://sinrega.org/2024-03-06-enabling-containers-gpu-macos/
I wonder if a similar technique can be used with Immich

[tdonovic]

Is virtio-gpu not an intel only thing?
This would require the underlying models to support the apple gpu bindings too no?
When I check on openclip it doesn't look like native mac support is there yet openai/CLIP#255

Establishing that would be the first step to whether its even worth trying to expose the gpu through docker.
Then I read something like https://chariotsolutions.com/blog/post/apple-silicon-gpus-docker-and-ollama-pick-two/, (only 3 months old) which says its not possible to expose the gpu and suggests running it locally.

I get the impression there is a lot of dev required elsewhere if its even possible before its worth spending time on the immich end.

[mertalev]

You would have to run the machine learning service directly outside of Docker. The instructions to do this are as follows:

1. Clone the immich repo
2. Navigate to the machine-learning folder
3. Add CoreMLExecutionProvider to the start of this list
4. Install Poetry
5. Install dependencies with poetry install --with cpu
6. Set the cache folder with export MACHINE_LEARNING_CACHE_FOLDER=~/.model-cache (you can set a different path here if you like)
7. Create the model cache folder with mkdir $MACHINE_LEARNING_CACHE_FOLDER
8. Run ./start.sh

It should then be listening on 3003.

Keep in mind that none of the models have been tested for CoreML compatibility, so you might need to experiment with different models to find one that works well.

[mertalev]

It seems like CoreML is misinterpreting this model.

When you say the others are "too big", what kind of error do you get with them?

[YazdanRa]

I get this message and it's still using CPU only and not the GPU on M1 Max

[10/27/24 17:12:31] INFO     Setting execution providers to ['CoreMLExecutionProvider', 'CPUExecutionProvider'], in descending order of preference                                                                                 
2024-10-27 17:12:31.551179 [W:onnxruntime:, coreml_execution_provider.cc:107 GetCapability] CoreMLExecutionProvider::GetCapability, number of partitions supported by CoreML: 101 number of nodes in the graph: 505 number of nodes supported by CoreML: 367                                                                           

2024-10-27 17:02:33.110054 [W:onnxruntime:, helper.cc:82 IsInputSupported] CoreML does not support input dim > 16384. Input:681, shape: {-1,25088}

Full Log:

(immich-machine-learning) ➜  machine-learning git:(main) ✗ ./start.sh
[10/27/24 17:08:45] INFO     Starting gunicorn 23.0.0                                                                                                                                                                              
[10/27/24 17:08:45] INFO     Listening at: http://[::]:3003 (57536)                                                                                                                                                                
[10/27/24 17:08:45] INFO     Using worker: app.config.CustomUvicornWorker                                                                                                                                                          
[10/27/24 17:08:45] INFO     Booting worker with pid: 57537                                                                                                                                                                        
[10/27/24 17:08:47] INFO     Started server process [57537]                                                                                                                                                                        
[10/27/24 17:08:47] INFO     Waiting for application startup.                                                                                                                                                                      
[10/27/24 17:08:47] INFO     Created in-memory cache with unloading after 300s of inactivity.                                                                                                                                      
[10/27/24 17:08:47] INFO     Initialized request thread pool with 10 threads.                                                                                                                                                      
[10/27/24 17:08:47] INFO     Application startup complete.                                                                                                                                                                         
[10/27/24 17:08:47] INFO     Loading detection model 'buffalo_l' to memory                                                                                                                                                         
[10/27/24 17:08:47] INFO     Setting execution providers to ['CoreMLExecutionProvider', 'CPUExecutionProvider'], in descending order of preference                                                                                 
2024-10-27 17:08:47.054241 [W:onnxruntime:, coreml_execution_provider.cc:107 GetCapability] CoreMLExecutionProvider::GetCapability, number of partitions supported by CoreML: 7 number of nodes in the graph: 153 number of nodes supported by CoreML: 129
[10/27/24 17:08:47] INFO     Loading recognition model 'buffalo_l' to memory                                                                                                                                                       
[10/27/24 17:08:47] INFO     Setting execution providers to ['CoreMLExecutionProvider', 'CPUExecutionProvider'], in descending order of preference                                                                                 
2024-10-27 17:08:47.457394 [W:onnxruntime:, helper.cc:82 IsInputSupported] CoreML does not support input dim > 16384. Input:681, shape: {-1,25088}
[10/27/24 17:12:22] INFO     Downloading visual model 'ViT-B-32__openai'. This may take a while.                                                                                                                                   
textual/tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 704/704 [00:00<00:00, 10.5MB/s]
textual/special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 472/472 [00:00<00:00, 6.21MB/s]
README.md: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 422/422 [00:00<00:00, 6.04MB/s]
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 196/196 [00:00<00:00, 3.21MB/s]
visual/preprocess_cfg.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 197/197 [00:00<00:00, 2.63MB/s]
textual/merges.txt: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 7.48MB/s]
.gitattributes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.75k/1.75k [00:00<00:00, 30.2MB/s]
textual/tokenizer.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.22M/2.22M [00:00<00:00, 18.2MB/s]
textual/vocab.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 862k/862k [00:00<00:00, 9.68MB/s]
model.onnx: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 254M/254M [00:05<00:00, 42.5MB/s]
model.onnx: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 351M/351M [00:08<00:00, 42.5MB/s]
Fetching 11 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:08<00:00,  1.29it/s]
[10/27/24 17:12:31] INFO     Loading visual model 'ViT-B-32__openai' to memory                                                                                                                                                     
[10/27/24 17:12:31] INFO     Setting execution providers to ['CoreMLExecutionProvider', 'CPUExecutionProvider'], in descending order of preference                                                                                 
2024-10-27 17:12:31.551179 [W:onnxruntime:, coreml_execution_provider.cc:107 GetCapability] CoreMLExecutionProvider::GetCapability, number of partitions supported by CoreML: 101 number of nodes in the graph: 505 number of nodes supported by CoreML: 367
[10/27/24 22:15:52] INFO     Handling signal: winch                                                                                                                                                                 

[mertalev]

It looks like it's using CoreML for the CLIP model, but not for the facial recognition model. The former is also split up into many parts, which is not ideal for GPU utilization.

[YazdanRa]

I'm putting together a simple PoC of an api in swift that uses Metal and Vision framework for facial recognition. Not sure how it'll go but theoretically if it follows the same schema as immich-machine-learning's api it should work just fine. Will post the progress here :)

[plsnotracking]

@YazdanRa feel free to hmu, I'd like help out/accelerate this.
CC: @mertalev