Colibri Core is a corpus analysis toolkit for discovering recurring patterns in large collections of text. Its approach goes beyond consecutive word sequences: it also models constructions in which particular positions are skipped or the gaps are allowed to vary in size.
The project is useful in corpus linguistics and computational language research, where the frequency of a pattern, its context and its relationships with other patterns can be as important as the exact words that occur together. Colibri Core represents text using compact numeric word classes to keep large pattern inventories practical.
The central pattern modeller can construct, inspect, query and manipulate models. Researchers can use its command-line tools for corpus processing or integrate the underlying functionality into their own applications. The project supports both direct pattern extraction and the analysis of relationships between extracted constructions.
This is free and open source software.
Key Features
- Extracts contiguous n-grams from encoded corpora.
- Models skipgrams containing fixed-size gaps.
- Models flexgrams with variable-length gaps.
- Uses compact numeric word-class encodings to limit memory consumption.
- Builds indexed and unindexed pattern models for different workloads.
- Calculates pattern frequencies and related corpus statistics.
- Queries individual patterns and retrieves occurrences from models.
- Computes associations and relationships between linguistic patterns.
- Supports model filtering, selection and manipulation.
- Offers command-line analysis tools and library interfaces.
- Includes Python bindings for corpus-analysis workflows.
- Filters candidate patterns using configurable minimum-occurrence thresholds.
- Limits extracted constructions by pattern length and gap characteristics.
- Can retain corpus occurrence positions in indexed models for contextual analysis.
- Cross-platform support – runs on Linux, FreeBSD, and macOS.
Website: github.com/proycon/colibri-core
Support:
Developer: Maarten van Gompel
License: GNU General Public License v3.0
Colibri Core is written in C++ and Python. Learn C++ with our recommended free books and free tutorials. Learn Python with our recommended free books and free tutorials.
Related Software
| C++ Natural Language Processing Tools | |
|---|---|
| text2vec | Framework with API for text analysis and natural language processing |
| Moses | Statistical machine translation system |
| TiMBL | Tilburg Memory-Based Learner |
| MITIE | MIT Information Extraction |
| MeTA | Modern C++ data sciences toolkit |
| Colibri Core | Efficient n-gram & skipgram modelling on text corpora |
| CRF++ | Yet Another CRF toolkit |
| BLLIP Parser | Statistical natural language parser |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |
Please read our Comment Policy before commenting.