Learn a subword tokenizer
Learn a subword tokenizer
Learn a byte-pair encoder from a corpus, save it, and split new words into its subwords (byte-pair encoders).
$ cat corpus.txt
low low low low lower lower newest newest newest widest
Learn the merges
Each merge joins the most frequent adjacent pair; the merges print in the order learned:
$ trex bpe train corpus.txt --merges 3
trex bpe: learned 3 merges from 56 bytes in 0.0s
l o
lo w
e s
Split words with it
let bpe = trex::bpe::Bpe::train(b"low low low low lower lower newest newest newest widest", 10);
assert_eq!(bpe.encode(b"lower").join(" "), "low er </w>");
let again = trex::bpe::Bpe::parse(&bpe.to_lines()).expect("a model");
assert_eq!(again.encode(b"lower"), bpe.encode(b"lower"));</w> marks a word’s end. The saved model is the merge list train prints, one pair a line,
and trex bpe encode --model FILE splits text with it.