SentencePiece trains language-independent subword tokenizers with BPE and unigram models, reversible encoding and text normalisation.
Read more
Tag: tokenization
tokenizers – fast and consistent text tokenisation in R
tokenizers converts text into words, sentences, n-grams, paragraphs and other token types with consistent functions for R analysis.
Read more
Ucto – rule-based Unicode text tokenizer
Ucto splits multilingual text into tokens and sentences, handles punctuation and normalisation, and supports FoLiA annotated output.
Read more