Support using OpenFst to compile HLG.

[csukuangfj]

It is to avoid OOM in determinizing LG when G is very large.

See also

• Exception when building HLG with large language model #132
• How to use large-scale language models in k2 k2#1010
• A error when compose H with LG k2#730
• Problem during preparing G #23

@Jarvan-Wang @wwx007121 Could you help test this PR? It should be able to handle large LMs and won't throw OOM as long as it does not throw OOM in kaldi as it is also using OpenFst to build LG.

---

I have tested the generated HLG with the pretrained model from
https://huggingface.co/Zengwei/icefall-asr-librispeech-lstm-transducer-stateless3-2022-09-28

It produces identical WER as the HLG generated by the master code for the first 30 utterances of test-clean and test-other.

One thing to note is that the file size of the resulting HLG is much smaller.

-rw-r--r-- 1 kuangfangjun root 806M Oct  8 10:51 data/lang_bpe_500/HLG.pt
-rw-r--r-- 1 kuangfangjun root 583M Oct  8 21:03 data/lang_bpe_500/HLG_fst.pt

HLG.pt is generated by the master code, while HLG_fst.pt is generated by this PR.

You have to install kaldifst to use this PR.

pip install kaldifst

or

conda install -c kaldifst kaldifst

The documentation for kaldifst is available at
https://k2-fsa.github.io/kaldifst/

[wangtiance]

Glad that I found this. Would it further help reducing the size of HLG if I do determinize and minimize after composing H with LG?

[wwx007121]

Glad that I found this. Would it further help reducing the size of HLG if I do determinize and minimize after composing H with LG?

This is caused by the size of the FST itself, and the smaller the FST will affect the recognition effect. So It is better to use Lattice instead of H to be composed with LG, if you want to decode with phone-level modeling/wfst

[wangtiance]

@wwx007121 I thought after minimizing the FST is equivalent as before, so it shouldn't affect recognition result?