Ucto is a rule-based text tokenizer intended for preparing material for linguistic analysis. It identifies word boundaries, separates punctuation and recognises sentence and paragraph structure, allowing subsequent tools to work with a consistent representation of the input.
The rules are supplied separately from the processing engine and can be adapted for particular languages or text collections. This makes Ucto suitable for corpus preprocessing where predictable token boundaries and reproducible rule sets matter more than training a neural model.
Ucto is part of the Language Machines family of linguistic tools and has been used in the Frog Dutch language processing pipeline. It can operate directly on text files and offers additional text normalization steps, including character encoding conversion and case transformation.
This is free and open source software.
Key Features
- Provides tokenization rule sets for English, Dutch, French, Italian, Turkish, Spanish, Portuguese and Swedish.
- Supports custom rule sets made from regular expressions and word lists.
- Identifies punctuation, abbreviations, measurements, currencies and common date or time forms.
- Recognises sentence boundaries, paragraph boundaries and paired quotation spans.
- Outputs UTF-8 text with Unicode NFC normalization.
- Can convert input text from other character encodings.
- Supports optional conversion to lowercase or uppercase.
- Can normalize ligatures during text preprocessing.
- Supports the FoLiA linguistic annotation format.
- Offers functionality suitable for automated corpus-processing pipelines.
- Handles explicit character and punctuation categories through reusable rules.
- Allows language-specific rules to be extended for specialized corpora.
- Supports whitespace-aware preprocessing for consistent sentence segmentation.
- Can be integrated with other components in the Language Machines toolchain.
Website: github.com/LanguageMachines/ucto
Support:
Developer: Maarten van Gompel, Ko van der Sloot and contributors
License: GNU General Public License v3.0

Ucto is written in C++. Learn C++ with our recommended free books and free tutorials.
Related Software
| C++ Natural Language Processing Tools | |
|---|---|
| SentencePiece | Language-independent tokenizer for neural text processing |
| Moses | Statistical machine translation system |
| CTranslate2 | Fast, memory-efficient inference for Transformer models |
| TiMBL | Tilburg Memory-Based Learner |
| Colibri Core | Efficient n-gram & skipgram modelling on text corpora |
| kenlm | Efficient toolkit for building statistical language models |
| CRF++ | Yet Another CRF toolkit |
| UDPipe | Trainable pipeline for tokenization, tagging and parsing |
| Marian | High-performance neural machine translation framework |
| Ucto | Rule-based, Unicode-aware tokenization for multilingual text |
Read our verdict in the software roundup.
LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |


Please read our Comment Policy before commenting.