[Help wanted] Support Flutter

[flutter-painter]

Could this possible using ffi ?
Ex : https://pub.dev/packages/onnxruntime

[nmfisher]

@flutter-painter I have already implemented this, was planning to open source it eventually but haven't got it around to it just yet. Send me an e-mail if you'd like.

[flutter-painter]

Hi @nmfisher,

Wonderful news,
I just sent an email,
Here is fine as well in case other aficionados wanna hop-in :)

[D-Idan]

Hi
Did you manage to make it happen?

[flutter-painter]

Almost, nick's repo is clear and his guidelines are complete.
I just need to build Onnx Runtime...
@nmfisher how about opening it as is ?

[D-Idan]

Let me know if I can help 🙂

[eschmidbauer]

also very interested in this- i have some flutter experience to test/help

[BrutalCoding]

@csukuangfj Am I right that the c-api only supports desktop platforms? I've tried porting your library over to Flutter today with ffi (https://docs.flutter.dev/platform-integration/android/c-interop) combined with ffigen (https://pub.dev/packages/ffigen).

I haven't pushed all my local changes because I haven't gotten it to work yet, but you can see the generated bindings here: https://github.com/BrutalCoding/aub.ai/blob/b16f7e5f9d317d83b38a5814afb1556c8de8a3d3/lib/aub_ai_bindings_generated.dart

I ran your the android build script for arm64-v8a successfully and they work fine when I'm using your native Android example apps. I can't get it to work in Flutter yet and would like some guidance.

Tree struc of aub.ai (not pushed to my repo, I'm trying to get it working locally first):
aub.ai/android/src/main/jniLibs/arm64-v8a
├── libcargs.so
├── libespeak-ng.so
├── libkaldi-decoder-core.so
├── libkaldi-native-fbank-core.so
├── libonnxruntime.so
├── libpiper_phonemize.so
├── libsherpa-onnx-c-api.so <- I tried with and without the C API.
├── libsherpa-onnx-core.so
├── libsherpa-onnx-fst.so
├── libsherpa-onnx-jni.so
├── libsherpa-onnx-kaldifst-core.so
└── libucd.so

1 directory, 12 files

When I look at your native Android examples, I noticed that Tts.kt is referring to 'sherpa-onnx-jni'.

 companion object {
        init {
            System.loadLibrary("sherpa-onnx-jni")
        }
    }

I've tried that one as well in Flutter by doing the following (aub_ai.dart):

...
return DynamicLibrary.open('libsherpa-onnx-jni.so');
...

But it fails to lookup the native symbols whenever I try to call functions from it. E.g. this fails:

// This function only relies on the onnxModel, the rest is hardcoded. 
 final Pointer<SherpaOnnxOfflineTtsConfig> ttsConfig = _getOfflineTtsConfig(
    pathToOnnxModel: pathToOnnxModel,
  );

  // Runtime error occurs on this line because it's failing to lookup the native symbol 'SherpaOnnxCreateOfflineTts'.
  final tts = sherpaOnnxBindings.SherpaOnnxCreateOfflineTts(ttsConfig);

Any tips?

I can solve it in a different way, by not using the c-api and instead use platform channels to execute native Kotlin code but having the c-api work for Android & iOS too would make long term maintenance easier and improves the dev experience (at least for me).

[nmfisher]

Hi @D-Idan @flutter-painter @eschmidbauer @BrutalCoding - sorry I've been really busy with other projects so haven't responded.

I've basically just flicked the switch to make this repository public:

https://github.com/nmfisher/flutter_sherpa_onnx

This is reasonably stable on iOS/Android (I have it running in production) and I've tested it informally on macos/Windows, seems no issues there.

As it says though, this is totally undocumented so you'll have to look at the code yourself to see how it works. Maybe @flutter-painter can submit a PR adding the notes I sent via email to the README which will give at least some explanation.

@BrutalCoding maybe this will help solve your issue.

[csukuangfj]

When I look at your native Android examples, I noticed that Tts.kt is referring to 'sherpa-onnx-jni'.

@BrutalCoding

sherpa-onnx-jni is for JVM, which is call by either java or kotlin. I am afraid you cannot use it with flutter. For flutter you can safely ignore libsherpa-onnx-jni.so.

---

I haven't pushed all my local changes because I haven't gotten it to work yet

Could you describe the issues you have? For instance, you can post error logs.

[BrutalCoding]

Hi @D-Idan @flutter-painter @eschmidbauer @BrutalCoding - sorry I've been really busy with other projects so haven't responded.

I've basically just flicked the switch to make this repository public:

https://github.com/nmfisher/flutter_sherpa_onnx

This is reasonably stable on iOS/Android (I have it running in production) and I've tested it informally on macos/Windows, seems no issues there.

As it says though, this is totally undocumented so you'll have to look at the code yourself to see how it works. Maybe @flutter-painter can submit a PR adding the notes I sent via email to the README which will give at least some explanation.

@BrutalCoding maybe this will help solve your issue.

I haven't dived into your code yet but I really appreciate it. Tried my best to get this to work on my own to no avail. Feels like an unhealthy hackathon for me last 24h. Trying to solve it from 9pm till 6:30am -> sleep -> again at 11am till I posted that comment (~6:30pm).

My day is almost up again but I'm certainly giving it a shot tmr with your repo, many thanks.

Could you describe the issues you have? For instance, you can post error logs.

My bad, I should've emphasized the actual error more, but it's just that Flutter is unable to find symbols in any of the .so files I've randomly picked (e.g. libsherpa-onnx-core.so). To confirm this, I used nm -g <filename>.so to find any symbols this way but there's nothing exposed (by design I guess).

Nonetheless, thanks for your quick reply. I will look into the solution that @nmfisher kindly shared, I don't expect any roadblockers anymore.

PS. Awesome CI/CD setup, it's what I imagined doing myself for a repo of mine but seeing it here is great to learn from. E.g. all binaries for each arch, the release versioning etc.

[flutter-painter]

Hi @nmfisher,
Here goes : nmfisher/sherpa_onnx_dart#2

[flutter-painter]

Hi,
Could dart ongoing support for native assets ease this integration process ?

[csukuangfj]

Hi, Could dart ongoing support for native assets ease this integration process ?

Sorry, I cannot understand it. Could you give more details?

[nmfisher]

I think this might be more relevant to my repository. Sent from my iPhoneOn 27 Feb 2024, at 9:21 AM, Fangjun Kuang ***@***.***> wrote: Hi, Could dart ongoing support for native assets ease this integration process ? Sorry, I cannot understand it. Could you give more details? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[BrutalCoding]

Hi, Could dart ongoing support for native assets ease this integration process ?

@flutter-painter While it would be nice, I think it would be better for @csukuangfj to continue focusing on existing work. The good news is that @csukuangfj already has makefiles, shell scripts, example apps for every platform and yeah- basically everything to build a fully working Flutter plugin.

It's probably a better idea for a (experienced/advanced) Flutter (package) dev to look into this project, unless sherpa-onnx is considered finished (it never is :P). While I can't share code or specific details, I can confirm that all the features of sherpa-onnx are working within Flutter. Think about TTS, ASR, VAD etc.

Sorry, I cannot understand it. Could you give more details?

@csukuangfj he is referring to https://github.com/dart-lang/native/tree/main/pkgs/native_assets_cli, which is a 1st party plugin (Google, or more specifically someone from the Dart team). It's the next way to interact with native code, basically a replacement for ffi and writing bindings for it. It's still experimental though so that's a 2nd reason that I discourage for you to jump in this potential rabbit hole.

While that is experimental, Flutter does already provide something for years that is robust:
Their 'ffi' package (plugin) combined with their 'ffigen' package, both found on https://pub.dev. Here is something that's easy to follow to allow native code in Flutter that was meant for pure C (note: Flutter uses Dart): https://docs.flutter.dev/platform-integration/android/c-interop

A Flutter dev could take a look at your native Android example projects and read through https://docs.flutter.dev/platform-integration/platform-channels?tab=type-mappings-kotlin-tab to get it working too.

TLDR: Many ways to achieve support of sherpa-onnx in Flutter, but it is challenging IMO if you want to be able to automate this for longevity / long term maintenance. Very do-able, it just takes a lot of time to set it up and test it for every platform.

I thought I'd just share some insights for anyone interested within the Flutter space.

[Web3Kev]

Hello., just thought I would bump this thread to see if anyone had progressed on a flutter package. I'm particularly interested in the tts function as I am porting a project that had been developed in c# and unity, and was using piper.

I am absolutely new to dart packages and how sherpa actually works but can I use @nmfisher 's repo which seems to have been tailored to speech recognition or should I give ffi a try ?

Thanks in advance.

[nmfisher]

@Web3Kev I'm working on the package at the moment to make it slightly more beginner friendly. Check back in a week or two to see if it's a bit easier to get up and running.

[csukuangfj]

Please see
#905 (comment)

[csukuangfj]

Here is the Dart package for sherpa-onnx
https://pub.dev/packages/sherpa_onnx

---

Pure Dart examples can be found at
https://github.com/k2-fsa/sherpa-onnx/tree/master/dart-api-examples

[beswarm]

the sherpa_onnx package seems do not support android and ios, any plans for mobile device support?

[csukuangfj]

the sherpa_onnx package seems do not support android and ios, any plans for mobile device support?

We need the community help to

• Refactor the current package to use the federated plugin
• Support android and ios, and the web

The current package is nearly 100 MB. pub.dev limits the maximum package size to 100MB.

[csukuangfj]

Closing via #1079

See also
https://pub.dev/packages/sherpa_onnx

Help to support the Web platform for Flutter is needed.

Note that we have already supported WebAssembly.