iPhone AMX Coprocessor -- Minimum Requirements?

[JKeddo95]

My iPhone 12 is about 10-15 times slower than my M1 Mac running whisper CPP. I am using the -DGGML_USE_ACCELERATE compiler flag, but I have a feeling that my phone just might not support it.

Apple's "AMX coprocessor" (AFAIK, what that compiler flag calls) doesn't seem to be well documented. I see lots of chatter online about it being used on Apple M1 CPU, but little talk of whether it's sibling chip A14 also has it. From the demo, it's obvious that a15 (iPhone 13) has it at least. Can anyone here confirm that iPhone 12 and older won't be able to use the special accelerator here?

[ggerganov]

iPhone 13 Mini with A15 with 4 threads is ~5 times slower than my MacBook M1 Pro at 8 threads in the Encoder.

Not sure about A14 - hopefully someone with that device can confirm the results.
Maybe it would be useful to post your Encoder time for a short (2-3 sec) audio.

In the meantime, I would also double-check that the the Accelerate framework is enabled.
If you remove the -DGGML_USE_ACCELERATE flag from the project does it should become even slower. Do you observe that?

[JKeddo95]

Thank you for the suggestion -- I will try that and let you know. Assuming I can get the time, I plan to upload a couple of benches tonight (iPhone 12 Pro (A14), iPhone 12 Pro Max (A14), Mac Pro 14 Base Model (M1 Pro), and an older iPad (Either A10 or A12, I need to double check).

What would be really interesting for me would be to see comparisons between M1 Pro VS M2 VS VS M2 Pro (coming soon) VS A16! Good data there could reveal where the bottleneck is for Apple Silicon: Coprocessor generation or thermal headroom. Who knows: Maybe Apple hasn't even updated the coprocessor this generation.

I just want to say thank you for your incredible work on this project. What you have done is nothing short of amazing!

[ggerganov]

The Encoder times should be reported in the console. See the videos in the whisper.objc example.
Thanks for the nice words! I really like how the project is going, and I hope we can make it even better!

[JKeddo95]

Hi there; I wasn't able to figure out how to get the bench to run on my iPhone 12 or iPad but I went ahead and uploaded the benches for my Macbook M1 Pro to the bench discussion thread. My results were about the same as yours.

I have attached my results below for my ad-hoc test of an iPhone 12 Pro with the input audio "I love transcriber apps":

CPU	DGGML_USE_ACCELERATE	OS	Model	Load	Mel	Sample	Enc.	Dec.	Total (ms)
A14	enabled	IOS 16.1	Base.en	3868	65	21	17649	715	46057
A14	disabled	IOS 16.1	Base.en	3790	48	25	54931	745	78793

--

EDIT: Please ignore the above benches. I had mistakenly thought the app was running in release mode when it was actually still running in debug mode. Please see the corrected benchmarks in my comment further below.

[ggerganov]

The 3868 ms load time most certainly means that you are running in Debug mode.
Make sure to change the project build type to Release:

[JKeddo95]

Wow! You are right, that was my issue! It is much faster now: Here are my new results:

CPU	DGGML_USE_ACCELERATE	OS	Model	Load	Mel	Sample	Enc.	Dec.	Total (ms)
A14	Release	IOS 16.1	Base.en	150	23	2	2447	112	2584