bpe Chinese tokens

BPE Algorithm of Algorithms to Understand in One Minute - KnowingFrom Deep Learning Natural Language ProcessingThe author of the WeChat public account zenRRan found one of the scripts of a machine translation when summarizing the analysis of the experimental data yesterday. The algorithm used in it is the BPE algorithm. It felt very high at first, because I always heard things with algorithm hats.Just think 666...https://zhuanlan.zhihu.com/p/38130825

bpe is very detailed as mentioned above.

The Chinese word code, currently mainly vocab.txt, is the 21128 version, which is the same version in Chinese_clip or Chinese_bert.

bpe English word encoding, currently the bpe_simple_vocab_16e6 version, 262145 version, used by both dalle and clip.