SentencePiece is a text tokenizer and detokenizer designed for natural language processing and neural network models. It converts text into subword units that can be used as input for machine translation, language modelling and text generation systems.
Unlike traditional tokenizers, SentencePiece operates directly on raw Unicode text without requiring language-specific word segmentation. This makes it particularly useful for multilingual applications and languages where words are not separated by spaces.
The software supports training custom tokenization models from text corpora, allowing developers to determine the vocabulary size and segmentation method appropriate for their applications. Its tokenization process preserves whitespace information, enabling the original text to be reconstructed from its tokens.
This is free and open source software.
Key Features
- Unsupervised training of subword tokenization models directly from raw text.
- Byte-Pair Encoding (BPE) and unigram language model tokenization algorithms.
- Reversible tokenization and detokenization preserving whitespace information.
- Language-independent processing of Unicode text.
- Configurable vocabulary sizes for neural language models.
- Subword regularization and BPE-dropout for generating alternative text segmentations during training.
- Text normalization with configurable Unicode normalization rules.
- Encode text as subword strings or numerical token identifiers.
- Support for user-defined symbols, control symbols and special tokens.
- Byte fallback for representing characters outside the trained vocabulary.
- Vocabulary restrictions and configurable character coverage.
- Multithreaded tokenization and model training.
- Self-contained model files storing vocabulary, normalization rules and segmentation information.
- Command-line tools and APIs for integrating tokenization into other applications.
- Lightweight runtime for performing tokenization with reduced dependencies.
- Cross-platform support – runs on Linux, macOS, and Windows.
Website: github.com/google/sentencepiece
Support:
Developer: Taku Kudo and contributors
License: Apache License 2.0
SentencePiece is written in C++. Learn C++ with our recommended free books and free tutorials.
Related Software
| C++ Natural Language Processing Tools | |
|---|---|
| text2vec | Framework with API for text analysis and natural language processing |
| Moses | Statistical machine translation system |
| TiMBL | Tilburg Memory-Based Learner |
| MITIE | MIT Information Extraction |
| MeTA | Modern C++ data sciences toolkit |
| Colibri Core | Efficient n-gram & skipgram modelling on text corpora |
| CRF++ | Yet Another CRF toolkit |
| BLLIP Parser | Statistical natural language parser |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |
Please read our Comment Policy before commenting.