KenLM is a toolkit for building and querying statistical n-gram language models. These models estimate the probability of word sequences and are widely used in applications such as automatic speech recognition and statistical machine translation.
The software focuses on efficient language model estimation and fast probability queries. It provides alternative data structures that allow developers to balance memory consumption against query performance, making it useful when working with large vocabularies and substantial text corpora.
KenLM supports the creation of language models from training text and their subsequent integration into decoding systems. It is designed for applications requiring repeated access to language model probabilities, including systems that evaluate many alternative word sequences.
This is free and open source software.
Key Features
- Estimate statistical n-gram language models from text corpora.
- Modified Kneser-Ney smoothing for estimating n-gram probabilities.
- Support for high-order n-gram models.
- Memory-efficient model estimation using disk-based algorithms.
- Read and query language models stored in ARPA format.
- Convert text-based language models into compact binary representations.
- Memory-mapped binary models for efficient access to language model data.
- Probing hash-table representation optimised for fast probability queries.
- Compact trie representation for reducing model memory consumption.
- Configurable trade-offs between memory usage and query speed.
- Language model filtering using corpus vocabularies, sentence vocabularies or phrases.
- Support for compressed input using gzip, bzip2 and xz formats.
- Stateful probability queries suitable for integration into decoding algorithms.
- Sentence scoring and word-sequence probability calculation.
- Programming interfaces for integrating language models into other applications.
- Python interface for loading models and evaluating sentences.
- Cross-platform support – runs on Linux, macOS, and Windows.
Website: github.com/kpu/kenlm
Support:
Developer: Kenneth Heafield and contributors
License: GNU Lesser General Public License v2.1
KenLM is written in C++. Learn C++ with our recommended free books and free tutorials.
Related Software
| C++ Natural Language Processing Tools | |
|---|---|
| text2vec | Framework with API for text analysis and natural language processing |
| Moses | Statistical machine translation system |
| TiMBL | Tilburg Memory-Based Learner |
| MITIE | MIT Information Extraction |
| MeTA | Modern C++ data sciences toolkit |
| Colibri Core | Efficient n-gram & skipgram modelling on text corpora |
| CRF++ | Yet Another CRF toolkit |
| BLLIP Parser | Statistical natural language parser |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |
Please read our Comment Policy before commenting.