[Idea]: Use Android NNAPI to accelerate inference on Android Devices

[Interpause]

This is just an idea for you. Most modern smartphones come with some form of AI accelerator. I am aware GGML-based projects like llama.cpp can compile and run on mobile devices, but there is probably performance left on the table. I think there is right now a gap for an mobile-optimized AI inference library with quantization support and the other tricks present in GGML. For reference: https://developer.android.com/ndk/guides/neuralnetworks

[Saghetti0]

Would love to see this as well!

[ggerganov]

If there is community help, we can try to add support for NNAPI. Currently, I don't have enough capacity to investigate this, but I think it is something interesting and can unlock many applications. Probably will look into this in the future and hoping there are some contributions in the meantime

[rhjdvsgsgks]

im trying to write a nnapi backend (well, you should not expect my work. because im a completely newbie. and mostly wont have any success). but after some document reading. i found that unlike cl or vk, nnapi didn't provide a way to use accelerated matrix multiply or some shader like stuff to compute something in gpu. the only things you can do with it is upload a graph of how layers connected (include operand and weight). so seems like it not very match the architecture llama.cpp current have? if no, please point me a backend using similar architecture so that i can have reference

[pax-k]

@ggerganov maybe it's worth checking NNAPI using ONNX runtime? WhisperRN runs smooth with CoreML, but on Android, even the tiny model is way too laggy to be usable on a budget device (for example Samsung a14, 4 GB RAM)

[Binozo]

@pax-k how do you define "laggy"? I am also investigating the performance on the Android side. My Samsung S22 is capable to transcribe a 30 second german voice message in about 3 seconds with the small whisper model.

I am also looking forward for the future because I am 100% sure the great Ai focus at Google will improve the Ai hardware for the next generations of Android phones.
It's better to be ready than being left behind.

With running in profile mode I could reduce the inference time by almost a second. I think this is acceptable.
(Running Flutter + onnxruntime + nnapi/xnnpack + 4 Thread processing)