Selection of utilities

Ucto – rule-based Unicode text tokenizer

Ucto is a rule-based text tokenizer intended for preparing material for linguistic analysis. It identifies word boundaries, separates punctuation and recognises sentence and paragraph structure, allowing subsequent tools to work with a consistent representation of the input.

The rules are supplied separately from the processing engine and can be adapted for particular languages or text collections. This makes Ucto suitable for corpus preprocessing where predictable token boundaries and reproducible rule sets matter more than training a neural model.

Ucto is part of the Language Machines family of linguistic tools and has been used in the Frog Dutch language processing pipeline. It can operate directly on text files and offers additional text normalization steps, including character encoding conversion and case transformation.

This is free and open source software.

Key Features

  • Provides tokenization rule sets for English, Dutch, French, Italian, Turkish, Spanish, Portuguese and Swedish.
  • Supports custom rule sets made from regular expressions and word lists.
  • Identifies punctuation, abbreviations, measurements, currencies and common date or time forms.
  • Recognises sentence boundaries, paragraph boundaries and paired quotation spans.
  • Outputs UTF-8 text with Unicode NFC normalization.
  • Can convert input text from other character encodings.
  • Supports optional conversion to lowercase or uppercase.
  • Can normalize ligatures during text preprocessing.
  • Supports the FoLiA linguistic annotation format.
  • Offers functionality suitable for automated corpus-processing pipelines.
  • Handles explicit character and punctuation categories through reusable rules.
  • Allows language-specific rules to be extended for specialized corpora.
  • Supports whitespace-aware preprocessing for consistent sentence segmentation.
  • Can be integrated with other components in the Language Machines toolchain.

Website: github.com/LanguageMachines/ucto
Support:
Developer: Maarten van Gompel, Ko van der Sloot and contributors
License: GNU General Public License v3.0

Options for Ucto

Ucto is written in C++. Learn C++ with our recommended free books and free tutorials.


Related Software

C++ Natural Language Processing Tools
SentencePieceLanguage-independent tokenizer for neural text processing
MosesStatistical machine translation system
CTranslate2Fast, memory-efficient inference for Transformer models
TiMBLTilburg Memory-Based Learner
Colibri CoreEfficient n-gram & skipgram modelling on text corpora
kenlmEfficient toolkit for building statistical language models
CRF++Yet Another CRF toolkit
UDPipeTrainable pipeline for tokenization, tagging and parsing
MarianHigh-performance neural machine translation framework
UctoRule-based, Unicode-aware tokenization for multilingual text

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted