Edgerunner

Universal AI inference library for mobile devices.

Website | Contact | Discord | Twitter | Docs

💡 Introduction

The purpose of Edgerunner is to facilitate easy integration of arbitrary AI models for consumer mobile devices (smartphones, laptops, tablets, wearables, etc.).

On-device inference tooling is fragmented. To run AI models efficiently on end-user devices, developers require a deep understanding of chip/device platforms (Apple, Qualcomm, MediaTek, etc.) and on-device AI frameworks ( TFLite, CoreML, QNN, etc.). Developers also need to maintain separate ML stacks for iOS and Android (which subdivides by chip platform).

Edgerunner aims to consolidate on-device AI frameworks, abstracting cross-platform complexities. This will be achieved through a runtime-agnostic API, which can load and interact with arbitrary instances of on-device AI models with just a few lines of code.

This repo is in its early stages of development (see Features and Support). There is lots of work to be done in order to achieve this vision, and your contributions will be important to make this happen!

 Kotlin bindings for Android can be found at edgerunner-android.

What makes Edgerunner unique?

Good question! Indeed there are other cross-platform libraries such as TFLite, ONNX, and ExecuTorch. Edgerunner will differentiate with:

• Shared libraries for platform specific SDKs are managed internally.
• Quicker and easier setup with sane defaults and no boilerplate code.
• Compatibility with any model format (.tflite, .onnx, .pte, etc).
• Easier configuration of delegates (CPU, GPU, NPU), applying desired hardware acceleration with a single API call.
• No manual memory allocation required (model input and output tensors are pre-allocated and accessed leading to copy-free inference - simply model.execute()).

🎁 Features

Feature	Status
Runtime-agnostic API	✅
Model loading	✅
Model execution	✅
Automatic framework detection	✅
Quantized models	⏳
Choose optimal execution at runtime	⏳
Java bindings	⏳
Objective-C bindings	⏳

Please request additional features through Github issues or on our Discord.

🔌 Support

OS

Linux	MacOS	Windows	Android	iOS
✅	✅	⏳	✅	⏳

Runtime Framework

TFLite	CoreML	Onnx	QNN	OpenVino	Ryzen AI	NeuroPilot
✅	⏳	⏳	⏳	⏳	⏳	⏳

Chip Vendor

	Apple	Qualcomm	MediaTek	Samsung	Intel	AMD	NVIDIA
CPU	✅	✅	✅	✅	✅	✅	⛔️
GPU	⏳	⏳	⏳	⏳	✅	✅	⏳
NPU	⏳	⏳	⏳	⛔️	⏳	⏳	⛔️

🛠 Building and installing

See the BUILDING document.

🕹 Usage

See examples for basic usage instructions.

🏆 Contributing

See the CONTRIBUTING document.

Join our Discord for discussing any issues.

📜 Licensing

See the LICENSING document.