Add streaming zipformer

[yaozengwei]

This PR adds recipe of streaming zipformer.

The following table compares performance of our current streaming models trained on full librispeech.

Model	Chunk	Right context	Model size	greedy-search	modified-beam-search	fast-beam-search	Training epochs
LSTM	-	-	85M	3.66 & 9.51	3.55 & 9.28	3.55 & 9.33	40
Conformer	320ms	-	79M	3.54 & 9.56	3.43 & 9.29	3.58 & 9.28	25
ConvEmformer	320ms	80ms	75M	3.6 & 9.11	3.49 & 8.85	3.6 & 8.95	30
Zipformer	320ms	-	70M	3.15 & 8.09	3.11 & 7.93	3.2 & 8.04	30
Conformer	640ms	-	79M	3.39 & 9.03	3.33 & 8.74	3.43 & 8.78	25
ConvEmformer	640ms	160ms	75M	3.3 & 8.71	3.26 & 8.56	3.27 & 8.58	30
Zipformer	640ms	-	70M	2.97 & 7.5	2.94 & 7.36	3.02 & 7.47	30

[ezerhouni]

@yaozengwei Thank you for this PR !
I am trying streaming zipformer (with my dataset) but I am getting Nan after few batches (~2500). Any idea, where it might comes from ?

[yaozengwei]

@yaozengwei Thank you for this PR ! I am trying streaming zipformer (with my dataset) but I am getting Nan after few batches (~2500). Any idea, where it might comes from ?

Could you first try to check the batch that starts to cause Nan? Maybe you could refer to https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/lstm_transducer_stateless3/train.py#L630.

If there is no problem with the data, you could run the training code with "--inf-check 1" before batche(~2500) that gets Nan, to check which module gets infinite.

[yaozengwei]

This PR could be merged first. I will create a new PR to add the document https://k2-fsa.github.io/icefall/recipes/Streaming-ASR/index.html.

[LoganLiu66]

When I using the modified_beam_search_lm_shallow_fusion decoding method with neural network LMs, following this, I get worse results compared with decoding without LM. And the partial decoding results are shown as follows:

1089-134686-0000-1189:	ref=['HE', 'HOPED', 'THERE', 'WOULD', 'BE', 'STEW', 'FOR', 'DINNER', 'TURNIPS', 'AND', 'CARROTS', 'AND', 'BRUISED', 'POTATOES', 'AND', 'FAT', 'MUTTON', 'PIECES', 'TO', 'BE', 'LADLED', 'OUT', 'IN', 'THICK', 'PEPPERED', 'FLOUR', 'FATTENED', 'SAUCE']
1089-134686-0000-1189:	hyp=['TOLD', 'PEOPLE', 'WERE', 'WHILEUT', 'YOU', 'TURN', 'TURN', 'WHILE', 'WERE', 'WHILEUT', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'WERE', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURN', 'PEOPLE', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'MI', 'TURN', 'VOICED', 'TURN', 'TURNTE', 'TURN', 'WERE', 'WHILEUT', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WEREAML', 'TURN', 'WERE', 'WHILEUT', 'TURNRU', 'NA', 'TURN', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'KEEP', 'VOICE', 'VOICE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'TURN', 'TURN', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WERE', 'WERE', 'WEREL', 'TURN', 'WERE', 'WEREUT', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILEAM', 'TURN', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'PEOPLE', 'TURN', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'TURN', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'ALWAYS', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE']
1089-134686-0001-1190:	ref=['STUFF', 'IT', 'INTO', 'YOU', 'HIS', 'BELLY', 'COUNSELLED', 'HIM']
1089-134686-0001-1190:	hyp=['TOLD', 'PEOPLE', 'WERE', 'WHILEUT', 'TOLD', 'TURN', 'TURN', 'WHILE', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURNTE', 'TURN', 'VOICE', 'VOICE', 'WHILE', 'WHILE', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'WERE', 'WHILEUT', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WHILEUT', 'WHILEUTUT', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'ALWAYS', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE']
1089-134686-0002-1191:	ref=['AFTER', 'EARLY', 'NIGHTFALL', 'THE', 'YELLOW', 'LAMPS', 'WOULD', 'LIGHT', 'UP', 'HERE', 'AND', 'THERE', 'THE', 'SQUALID', 'QUARTER', 'OF', 'THE', 'BROTHELS']
1089-134686-0002-1191:	hyp=['PER', 'PER', 'WERE', 'WHILEUT', 'WHILEUT', 'KEEP', 'WEREUT', 'WERE', 'WHILEUT', 'TURN', 'WERE', 'TURN', 'WERE', 'VOICE', 'PER', 'TURN', 'TURN', 'WERE', 'WERE', 'WERE', 'TURN', 'STRANGE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREL', 'STRANGE', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURNL', 'TURN', 'WERE', 'TURNL', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'WERE', 'TURNL', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREL', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'PER', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILEUTUT', 'WERE', 'WHILEUTUT', 'WERE', 'WHILEUT', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE']
1089-134686-0003-1192:	ref=['HELLO', 'BERTIE', 'ANY', 'GOOD', 'IN', 'YOUR', 'MIND']
1089-134686-0003-1192:	hyp=['PER', 'PER', 'WERE', 'WHILEUT', 'WHILEUT', 'KEEP', 'WERE', 'WHILEUT', 'KEEPIL', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'STRANGE', 'TURN', 'TURN', 'TURNRU', 'TURN', 'TURNRU', 'TURN', 'YOU', 'TURN', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE']
1089-134686-0004-1193:	ref=['NUMBER', 'TEN', 'FRESH', 'NELLY', 'IS', 'WAITING', 'ON', 'YOU', 'GOOD', 'NIGHT', 'HUSBAND']
1089-134686-0004-1193:	hyp=['PER', 'PEOPLE', 'WERE', 'WHILE', 'PEOPLE', 'TURN', 'TURN', 'TURN', 'TURN', 'YOU', 'TURN', 'YOU', 'TURN', 'TURN', 'WERE', 'TURNUT', 'WERE', 'WHILEUT', 'TURNRUOR', 'TURN', 'TURN', 'MI', 'TURN', 'STRANGE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'TURN', 'WERE', 'TURN', 'STRANGE', 'TURN', 'WEREAML', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'WERE', 'WHILEUT', 'YOU', 'RI', 'RIUT', 'WERE', 'WHILE', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURNTE', 'VOICE', 'VOICE', 'WHILE', 'BEEN', 'SMALL', 'PER', 'VOICE']
1089-134686-0005-1194:	ref=['THE', 'MUSIC', 'CAME', 'NEARER', 'AND', 'HE', 'RECALLED', 'THE', 'WORDS', 'THE', 'WORDS', 'OF', "SHELLEY'S", 'FRAGMENT', 'UPON', 'THE', 'MOON', 'WANDERING', 'COMPANIONLESS', 'PALE', 'FOR', 'WEARINESS']
1089-134686-0005-1194:	hyp=['TOLD', 'PEOPLE', 'WEREUT', 'WERE', 'WHILEUT', 'TOLD', 'TURN', 'WERE', 'WHILEUT', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'TURNUT', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'TURN', 'PEOPLE', 'TURN', 'TURN', 'TURN', 'TURN', 'MID', 'TURNILD', 'TURNILD', 'TURN', 'TURN', 'TURN', 'TURN', 'MI', 'TURN', 'VOICEOUR', 'TURN', 'TURN', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREAMUT', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'VOICEQUI', 'TURN', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'WERE', 'WHILEAM', 'TURNRU', 'NA', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURNL', 'TURN', 'VOICE', 'VOICE', 'TURN', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUTTE', 'VOICE', 'VOICETE', 'VOICE', 'VOICE', 'VOICE', 'VOICE', 'VOICE', 'VOICE', 'VOICE', 'VOICE', 'VOICE', 'VOICE', 'VOICETE', 'VOICE', 'VOICE', 'WHILE', 'TURN', 'TURNTE', 'TURN', 'MI', 'TURN', 'TURN', 'MI', 'TURN', 'WHILE', 'PEOPLE', 'TURN', 'TURNTE', 'VOICE', 'VOICE', 'TURN', 'TURN', 'WERE', 'WEREUTTE', 'TURN', 'MI', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'TURN', 'WERE', 'TURN', 'WERE', 'VOICE', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'VOICE', 'VOICETE', 'VOICE', 'MI', 'VOICE', 'VOICE', 'TURN', 'TURN', 'EN', 'VOICE', 'VOICE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'YOU', 'VOICE', 'TURN', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE']
1089-134686-0006-1195:	ref=['THE', 'DULL', 'LIGHT', 'FELL', 'MORE', 'FAINTLY', 'UPON', 'THE', 'PAGE', 'WHEREON', 'ANOTHER', 'EQUATION', 'BEGAN', 'TO', 'UNFOLD', 'ITSELF', 'SLOWLY', 'AND', 'TO', 'SPREAD', 'ABROAD', 'ITS', 'WIDENING', 'TAIL']
1089-134686-0006-1195:	hyp=['TOLD', 'PEOPLE', 'WERE', 'WHILEUT', 'TOLD', 'TURN', 'TURN', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'PEOPLE', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'VOICETE', 'VOICE', 'VOICE', 'RI', 'TURN', 'TURN', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'VOICEQUID', 'VOICE', 'TURND', 'TURN', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'PEOPLE', 'TURN', 'TURNTE', 'TURN', 'VOICE', 'WERE', 'TURN', 'WERE', 'TURNL', 'TURN', 'WERE', 'TURN', 'WEREAM', 'PER', 'TURN', 'VOICE', 'VOICE', 'TURN', 'VOICE', 'TURN', 'TURN', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILEQUI', 'TURN', 'TURN', 'TURN', 'MI', 'TURN', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'TURN', 'WERE', 'WHILEUT', 'YOU', 'VOICE', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'PEOPLE', 'TURN', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'KEEP', 'VOICE', 'VOICETE', 'TURN', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'WEREUTLUT', 'WERE', 'WHILEUT', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREL', 'TURN', 'WERE', 'WEREUTUT', 'WERE', 'WHILESHSHUT', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILE', 'WERE', 'WHILEUT', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILEUT', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUT', 'WERE', 'WHILEUT', 'WHILEUT', 'PER', 'WHILE', 'WERE', 'TURN', 'WERE', 'WERE', 'WERE', 'WERE', 'WERE', 'WEREUTTE', 'VOICE', 'VOICE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE', 'WHILE']
1089-134686-0007-1196:	ref=['A', 'COLD', 'LUCID', 'INDIFFERENCE', 'REIGNED', 'IN', 'HIS', 'SOUL']

What are the possible reasons for this output

[csukuangfj]

All hyps are the same. Have you changed any code?

[marcoyang1998]

What's your lm scale? @JiawangLiu

[LoganLiu66]

Decoding results given by fast beam search、modified beam search are as expected. Then I just download the LM weights from this and run the following commond to decode, nothing else has been changed.

for lm_scale in $(seq 0.15 0.01 0.38); do
    for beam_size in 4 8 12; do
        ./pruned_transducer_stateless7_streaming/decode.py \
            --epoch 99 \
            --avg 1 \
            --use-averaged-model False \
            --beam-size $beam_size \
            --exp-dir ./pruned_transducer_stateless7_streaming/exp-large-LM \
            --max-duration 600 \
            --decode-chunk-len 32 \
            --decoding-method modified_beam_search_lm_shallow_fusion \
            --use-shallow-fusion 1 \
            --lm-type rnn \
            --lm-exp-dir rnn_lm/exp \
            --lm-epoch 99 \
            --lm-scale $lm_scale \
            --lm-avg 1 \
            --rnn-lm-embedding-dim 2048 \
            --rnn-lm-hidden-dim 2048 \
            --rnn-lm-num-layers 3 \
            --lm-vocab-size 500
    done
done

[ezerhouni]

@JiawangLiu Could you check if the bpe model is the same, it looks to me as a mismatch between the bpe from the RNNLM and the Zipformer

[LoganLiu66]

I can't find the bpe model in RNNLM, I get my bpe_500 model by running the ./prepare.sh

[ezerhouni]

@JiawangLiu Then very most likely it is not the same as the RNN-LM one. To be honest, I don't recall which one I used for training the RNN-LM