Ucto – advanced rule-based unicode-aware tokenizer

Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation.

Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor.

This is free and open source software.

Features include:

• Comes with tokenization rules for English, Dutch, French, Italian, and Swedish; easily extendible to other languages. Rules consists of regular expressions and lists. They are packaged separately as uctodata.
• Recognizes dates, times, units, currencies, abbreviations.
• Recognizes paired quote spans, sentences, and paragraphs.
• Produces UTF8 encoding and NFC output normalization, optionally accepts other encodings as input.
• Optional conversion to all lowercase or uppercase.
• Supports FoLiA XML.

Website: languagemachines.github.io/ucto
Support:
Developer: Maarten van Gompel, Ko van der Sloot, Folgert Karsdorp, Iris Hendrickx
License: MIT License

Ucto is written in C++. Learn C++ with our recommended free books and free tutorials.

Explore Linux
	Best Free and Open Source Software - A huge collection of recommended free and open source software covering every major category.
	In-depth Linux Software Reviews - Detailed assessments of Linux software covering features, usability and overall quality.
	The Big List of Active Linux Distros - An extensive collection of actively developed Linux distributions.
	Open Source Alternatives - Alternatives to software and services from Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle and others.
	Essential Linux System Tools - Indispensable utilities for monitoring, maintaining and managing Linux systems.
	Linux Productivity Tools - Useful utilities for improving productivity and streamlining everyday tasks.
	Alternatives to Popular CLI Tools - Modern alternatives to familiar command-line utilities supplied with Linux systems.
	Awesome Free Linux Game Tools - Tools, utilities and companion software that improve gaming on Linux.
	Machine Learning on Linux - Machine learning and deep learning software, self-hosted applications and practical tools.
	Saving Money with Linux - Practical ways to reduce electricity consumption and lower computing costs with Linux.
	Linux Candy - Fun, entertaining and sometimes delightfully quirky software from the lighter side of Linux.