Skip to content

Commit 2c2926a

Browse files
csukuangfjcsukuangfj
authored
Add C++ runtime for Matcha-TTS (#1627)
1 parent 5c2cc48 commit 2c2926a

33 files changed

+1397
-86
lines changed

‎.github/scripts/test-offline-tts.sh

+34
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,40 @@ which $EXE
1818
# test waves are saved in ./tts
1919
mkdir ./tts
2020

21+
log "------------------------------------------------------------"
22+
log "matcha-icefall-zh-baker"
23+
log "------------------------------------------------------------"
24+
curl -O -SL https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/matcha-icefall-zh-baker.tar.bz2
25+
tar xvf matcha-icefall-zh-baker.tar.bz2
26+
rm matcha-icefall-zh-baker.tar.bz2
27+
28+
curl -SL -O https://github.com/k2-fsa/sherpa-onnx/releases/download/vocoder-models/hifigan_v2.onnx
29+
30+
$EXE \
31+
--matcha-acoustic-model=./matcha-icefall-zh-baker/model-steps-3.onnx \
32+
--matcha-vocoder=./hifigan_v2.onnx \
33+
--matcha-lexicon=./matcha-icefall-zh-baker/lexicon.txt \
34+
--matcha-tokens=./matcha-icefall-zh-baker/tokens.txt \
35+
--matcha-dict-dir=./matcha-icefall-zh-baker/dict \
36+
--num-threads=2 \
37+
--debug=1 \
38+
--output-filename=./tts/matcha-baker-zh-1.wav \
39+
'小米的使命是，始终坚持做"感动人心、价格厚道"的好产品，让全球每个人都能享受科技带来的美好生活'
40+
41+
$EXE \
42+
--matcha-acoustic-model=./matcha-icefall-zh-baker/model-steps-3.onnx \
43+
--matcha-vocoder=./hifigan_v2.onnx \
44+
--matcha-lexicon=./matcha-icefall-zh-baker/lexicon.txt \
45+
--matcha-tokens=./matcha-icefall-zh-baker/tokens.txt \
46+
--matcha-dict-dir=./matcha-icefall-zh-baker/dict \
47+
--num-threads=2 \
48+
--debug=1 \
49+
--output-filename=./tts/matcha-baker-zh-2.wav \
50+
"当夜幕降临，星光点点，伴随着微风拂面，我在静谧中感受着时光的流转，思念如涟漪荡漾，梦境如画卷展开，我与自然融为一体，沉静在这片宁静的美丽之中，感受着生命的奇迹与温柔。"
51+
52+
rm hifigan_v2.onnx
53+
rm -rf matcha-icefall-zh-baker
54+
2155
log "------------------------------------------------------------"
2256
log "vits-piper-en_US-amy-low"
2357
log "------------------------------------------------------------"

‎.github/scripts/test-python.sh

+20
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,26 @@ mkdir ./tts
269269

270270
log "vits-ljs test"
271271

272+
curl -O -SL https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/matcha-icefall-zh-baker.tar.bz2
273+
tar xvf matcha-icefall-zh-baker.tar.bz2
274+
rm matcha-icefall-zh-baker.tar.bz2
275+
276+
curl -SL -O https://github.com/k2-fsa/sherpa-onnx/releases/download/vocoder-models/hifigan_v2.onnx
277+
278+
python3 ./python-api-examples/offline-tts.py \
279+
--matcha-acoustic-model=./matcha-icefall-zh-baker/model-steps-3.onnx \
280+
--matcha-vocoder=./hifigan_v2.onnx \
281+
--matcha-lexicon=./matcha-icefall-zh-baker/lexicon.txt \
282+
--matcha-tokens=./matcha-icefall-zh-baker/tokens.txt \
283+
--tts-rule-fsts=./matcha-icefall-zh-baker/phone.fst,./matcha-icefall-zh-baker/date.fst,./matcha-icefall-zh-baker/number.fst \
284+
--matcha-dict-dir=./matcha-icefall-zh-baker/dict \
285+
--output-filename=./tts/test-matcha.wav \
286+
"某某银行的副行长和一些行政领导表示，他们去过长江和长白山; 经济不断增长。2024年12月31号，拨打110或者18920240511。123456块钱。"
287+
288+
rm -rf matcha-icefall-zh-baker
289+
rm hifigan_v2.onnx
290+
291+
272292
curl -LS -O https://huggingface.co/csukuangfj/vits-ljs/resolve/main/vits-ljs.onnx
273293
curl -LS -O https://huggingface.co/csukuangfj/vits-ljs/resolve/main/lexicon.txt
274294
curl -LS -O https://huggingface.co/csukuangfj/vits-ljs/resolve/main/tokens.txt

‎.github/workflows/linux.yaml

+17-13
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,23 @@ jobs:
149149
name: release-${{ matrix.build_type }}-with-shared-lib-${{ matrix.shared_lib }}-with-tts-${{ matrix.with_tts }}
150150
path: install/*
151151

152+
- name: Test offline TTS
153+
if: matrix.with_tts == 'ON'
154+
shell: bash
155+
run: |
156+
du -h -d1 .
157+
export PATH=$PWD/build/bin:$PATH
158+
export EXE=sherpa-onnx-offline-tts
159+
160+
.github/scripts/test-offline-tts.sh
161+
du -h -d1 .
162+
163+
- uses: actions/upload-artifact@v4
164+
if: matrix.with_tts == 'ON'
165+
with:
166+
name: tts-generated-test-files-${{ matrix.build_type }}-${{ matrix.shared_lib }}-with-tts-${{ matrix.with_tts }}
167+
path: tts
168+
152169
- name: Test offline Moonshine
153170
if: matrix.build_type != 'Debug'
154171
shell: bash
@@ -309,16 +326,7 @@ jobs:
309326
.github/scripts/test-offline-whisper.sh
310327
du -h -d1 .
311328
312-
- name: Test offline TTS
313-
if: matrix.with_tts == 'ON'
314-
shell: bash
315-
run: |
316-
du -h -d1 .
317-
export PATH=$PWD/build/bin:$PATH
318-
export EXE=sherpa-onnx-offline-tts
319329
320-
.github/scripts/test-offline-tts.sh
321-
du -h -d1 .
322330
323331
- name: Test online paraformer
324332
shell: bash
@@ -367,8 +375,4 @@ jobs:
367375
overwrite: true
368376
file: sherpa-onnx-*.tar.bz2
369377

370-
- uses: actions/upload-artifact@v4
371-
with:
372-
name: tts-generated-test-files-${{ matrix.build_type }}-${{ matrix.shared_lib }}-with-tts-${{ matrix.with_tts }}
373-
path: tts
374378

‎.github/workflows/macos.yaml

+9-9
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,15 @@ jobs:
121121
otool -L build/bin/sherpa-onnx
122122
otool -l build/bin/sherpa-onnx
123123
124+
- name: Test offline TTS
125+
if: matrix.with_tts == 'ON'
126+
shell: bash
127+
run: |
128+
export PATH=$PWD/build/bin:$PATH
129+
export EXE=sherpa-onnx-offline-tts
130+
131+
.github/scripts/test-offline-tts.sh
132+
124133
- name: Test offline Moonshine
125134
if: matrix.build_type != 'Debug'
126135
shell: bash
@@ -226,15 +235,6 @@ jobs:
226235
227236
.github/scripts/test-kws.sh
228237
229-
- name: Test offline TTS
230-
if: matrix.with_tts == 'ON'
231-
shell: bash
232-
run: |
233-
export PATH=$PWD/build/bin:$PATH
234-
export EXE=sherpa-onnx-offline-tts
235-
236-
.github/scripts/test-offline-tts.sh
237-
238238
- name: Test online paraformer
239239
shell: bash
240240
run: |

‎python-api-examples/offline-tts-play.py

+85-8
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
1212
Usage:
1313
14-
Example (1/3)
14+
Example (1/4)
1515
1616
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-amy-low.tar.bz2
1717
tar xf vits-piper-en_US-amy-low.tar.bz2
@@ -23,7 +23,7 @@
2323
--output-filename=./generated.wav \
2424
"Today as always, men fall into two groups: slaves and free men. Whoever does not have two-thirds of his day for himself, is a slave, whatever he may be: a statesman, a businessman, an official, or a scholar."
2525
26-
Example (2/3)
26+
Example (2/4)
2727
2828
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-zh-aishell3.tar.bz2
2929
tar xvf vits-zh-aishell3.tar.bz2
@@ -37,7 +37,7 @@
3737
--output-filename=./liubei-21.wav \
3838
"勿以恶小而为之，勿以善小而不为。惟贤惟德，能服于人。122334"
3939
40-
Example (3/3)
40+
Example (3/4)
4141
4242
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/sherpa-onnx-vits-zh-ll.tar.bz2
4343
tar xvf sherpa-onnx-vits-zh-ll.tar.bz2
@@ -53,6 +53,24 @@
5353
--output-filename=./test-2.wav \
5454
"当夜幕降临，星光点点，伴随着微风拂面，我在静谧中感受着时光的流转，思念如涟漪荡漾，梦境如画卷展开，我与自然融为一体，沉静在这片宁静的美丽之中，感受着生命的奇迹与温柔。2024年5月11号，拨打110或者18920240511。123456块钱。"
5555
56+
Example (4/4)
57+
58+
curl -O -SL https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/matcha-icefall-zh-baker.tar.bz2
59+
tar xvf matcha-icefall-zh-baker.tar.bz2
60+
rm matcha-icefall-zh-baker.tar.bz2
61+
62+
curl -SL -O https://github.com/k2-fsa/sherpa-onnx/releases/download/vocoder-models/hifigan_v2.onnx
63+
64+
python3 ./python-api-examples/offline-tts-play.py \
65+
--matcha-acoustic-model=./matcha-icefall-zh-baker/model-steps-3.onnx \
66+
--matcha-vocoder=./hifigan_v2.onnx \
67+
--matcha-lexicon=./matcha-icefall-zh-baker/lexicon.txt \
68+
--matcha-tokens=./matcha-icefall-zh-baker/tokens.txt \
69+
--tts-rule-fsts=./matcha-icefall-zh-baker/phone.fst,./matcha-icefall-zh-baker/date.fst,./matcha-icefall-zh-baker/number.fst \
70+
--matcha-dict-dir=./matcha-icefall-zh-baker/dict \
71+
--output-filename=./test-matcha.wav \
72+
"某某银行的副行长和一些行政领导表示，他们去过长江和长白山; 经济不断增长。2024年12月31号，拨打110或者18920240511。123456块钱。"
73+
5674
5775
You can find more models at
5876
https://github.com/k2-fsa/sherpa-onnx/releases/tag/tts-models
@@ -84,14 +102,11 @@
84102
sys.exit(-1)
85103

86104

87-
def get_args():
88-
parser = argparse.ArgumentParser(
89-
formatter_class=argparse.ArgumentDefaultsHelpFormatter
90-
)
91-
105+
def add_vits_args(parser):
92106
parser.add_argument(
93107
"--vits-model",
94108
type=str,
109+
default="",
95110
help="Path to vits model.onnx",
96111
)
97112

@@ -124,6 +139,60 @@ def get_args():
124139
help="Path to the dict directory for models using jieba",
125140
)
126141

142+
143+
def add_matcha_args(parser):
144+
parser.add_argument(
145+
"--matcha-acoustic-model",
146+
type=str,
147+
default="",
148+
help="Path to model.onnx for matcha",
149+
)
150+
151+
parser.add_argument(
152+
"--matcha-vocoder",
153+
type=str,
154+
default="",
155+
help="Path to vocoder for matcha",
156+
)
157+
158+
parser.add_argument(
159+
"--matcha-lexicon",
160+
type=str,
161+
default="",
162+
help="Path to lexicon.txt for matcha",
163+
)
164+
165+
parser.add_argument(
166+
"--matcha-tokens",
167+
type=str,
168+
default="",
169+
help="Path to tokens.txt for matcha",
170+
)
171+
172+
parser.add_argument(
173+
"--matcha-data-dir",
174+
type=str,
175+
default="",
176+
help="""Path to the dict directory of espeak-ng. If it is specified,
177+
--matcha-lexicon and --matcha-tokens are ignored""",
178+
)
179+
180+
parser.add_argument(
181+
"--matcha-dict-dir",
182+
type=str,
183+
default="",
184+
help="Path to the dict directory for models using jieba",
185+
)
186+
187+
188+
def get_args():
189+
parser = argparse.ArgumentParser(
190+
formatter_class=argparse.ArgumentDefaultsHelpFormatter
191+
)
192+
193+
add_vits_args(parser)
194+
add_matcha_args(parser)
195+
127196
parser.add_argument(
128197
"--tts-rule-fsts",
129198
type=str,
@@ -313,6 +382,14 @@ def main():
313382
dict_dir=args.vits_dict_dir,
314383
tokens=args.vits_tokens,
315384
),
385+
matcha=sherpa_onnx.OfflineTtsMatchaModelConfig(
386+
acoustic_model=args.matcha_acoustic_model,
387+
vocoder=args.matcha_vocoder,
388+
lexicon=args.matcha_lexicon,
389+
tokens=args.matcha_tokens,
390+
data_dir=args.matcha_data_dir,
391+
dict_dir=args.matcha_dict_dir,
392+
),
316393
provider=args.provider,
317394
debug=args.debug,
318395
num_threads=args.num_threads,

0 commit comments

Comments
 (0)