Android maintained its position as the leading mobile operating system worldwide in the fourth quarter of 2023 with a market share of 70.1 percent . Qualcomm is No.1 mobile SoC semiconductor company in our planet currently.
QNN(Qualcomm Neural Network, aka Qualcomm AI Engine Direct) SDK is verified to work with the following versions of the ML frameworks:
- TensorFlow: tf-1.15.0, or tf-2.10.1
- TFLite: tflite-2.3.0
- PyTorch: torch-1.13.1
- ONNX: onnx-1.11.0
The Qualcomm® AI Engine Direct architecture is designed to be modular and allows for clean separation in the software for different hardware cores/accelerators such as the CPU, GPU and DSP that are designated as backends. Learn more about Qualcomm® AI Engine Direct backends here.
The Qualcomm® AI Engine Direct backends for different hardware cores/accelerators are compiled into individual core-specific libraries that come packaged with the SDK.
One of the key highlights of Qualcomm® AI Engine Direct is that it provides a unified API to delegate operations such as graph creation and execution across all hardware accelerator backends. This allows users to treat Qualcomm® AI Engine Direct as a hardware abstraction API and port applications easily to different cores.
The Qualcomm® AI Engine Direct API is designed to support an efficient execution model with capabilities such as graph optimizations to be taken care of internally. At the same time however, it leaves out broader functionality such as model parsing and network partitioning to higher level frameworks.
Qualcomm® AI Engine Direct API and the associated software stack provides all the constructs required by an application to construct, optimize and execute network models on the desired hardware accelerator core. Key constructs are illustrated by the Qualcomm AI Engine Direct Components - High Level View diagram.
The llama.cpp QNN backend is intented to support Qualcomm mobile SoC firstly.
-
2024.5.28
- re-launch activity of PR in upstream ggml community
-
2024.4.26
- refine PR according to coding stye and pricinples of upstream ggml community
- add command line test using test-backend-ops.cpp
- refine PR according comments from reviewer
-
2024.4.24
- a very beginning PR to upstream ggml community
- data path works fine as expected with whisper.cpp and llama.cpp using QNN backend and verified on both low-end and high-end Android phones based on Qualcomm mobile SoC
- Support OPs
- GGML_OP_ADD
- GGML_OP_MUL
- GGML_OP_MUL_MAT
-
2024.3.29
| OS | Status | Verified |
|---|---|---|
| Android | Support | Android 10, Android 14 |
| Windows over ARM | TBD | TBD |
Verified devices
| Qualcom mobile SoC | Status | Verified Vendor |
|---|---|---|
| Qualcomm SM8650-AB Snapdragon 8 Gen 3 | Support | Xiaomi 14 |
| Qualcomm low-end mobile SoC Series | Support | Vivo |
TBD
Any mainstream Android phone based on Qualcomm's mobile SoC should be supported by llama.cpp + QNN. Qualcomm SM8650-AB Snapdragon 8 Gen 3 based Android phone is preferred.
- for QNN backend developers, download and install QNN SDK from Qualcomm offcial website
https://qpm.qualcomm.com/#/main/tools/details/qualcomm_ai_engine_direct
https://developer.qualcomm.com/software/hexagon-dsp-sdk/tools
the default installation path is /opt/qcom/aistack/qnn/2.20.0.240223/
- for programmers, using test-backend-ops.cpp to verify QNN backend on Qualcomm mobile SoC based Android phone
cd core/ggml/llamacpp/tests/ggml-qnn/
./build-ggml-qnn.sh
./run-ggml-qnn.sh
pls refer to README.md
TBD
TBD
Please add the [ggml-qnn] prefix/tag in issues/PRs titles to help the community check/address them without delay.
-
only support FP32 / FP16 and the input and output tensors must be of the same data type
-
multithreading not working with QNN GPU&HTP (aka DSP) backend
-
QNN's RPC feature(which useful for QNN HTP(aka DSP) backend) not used
-
multi QNN backend(CPU/GPU/DSP) simultaneously not support