UnicodeEncodeError throw by export_bpe_vocab.py

[sheldonrobinson]

Problem

Encountered following error running export_bpe_vocab.py:

sherpa-onnx\scripts\export_bpe_vocab.py --bpe-model bpe.model
Traceback (most recent call last):
  File "sherpa-onnx\scripts\export_bpe_vocab.py", line 70, in <module>
    main()
  File "sherpa-onnx\scripts\export_bpe_vocab.py", line 65, in main
    vfile.write(f"{v}\t{sp.get_score(id)}\n")
  File "C:\Programs\Python\Python312\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u2581' in position 0: character maps to <undefined>

Solution

Add encoding utf-8 when opening the vfile