Use Emformer as RNN-T encoder.

[csukuangfj]

We import Emformer from torchaudio directly at present and use it to replace the Conformer model for the current RNN-T setup.

[csukuangfj]

Here is a recording about the streaming decoding using RNN-T greedy search with Emformer.

Screen.Recording.2022-04-01.at.7.43.04.PM.mov

The number of model parameters is: 65390556

The WERs using --epoch 29 --avg 6 --decoding_method modified_beam_search are 4.58/11.54 (test-clean/test-other),
which is very close to the one listed in https://github.com/pytorch/audio/tree/main/examples/asr/emformer_rnnt, i.e., 4.56 for test-clean (note that one is trained for 120 epochs using 32 GPUs, while this one is trained using 8 GPUs).

[pzelasko]

Great news!

[csukuangfj]

I am merging it to the streaming branch first.

@yaozengwei will work on the streaming decoding part with beam search (--max-sym-per-frame=1)