CTranslate2 is an inference engine for Transformer models, designed to execute neural networks efficiently on CPUs and GPUs. It is suitable for applications involving machine translation, speech recognition, text generation and other natural language processing tasks.
The software uses a specialised execution runtime rather than relying on a general-purpose deep learning framework. Models trained using compatible frameworks can be converted into an optimised representation for deployment.
CTranslate2 focuses on reducing memory consumption and improving inference speed while retaining the functionality required for production applications. It supports several Transformer architectures, making it useful for deploying both sequence-to-sequence and language generation models.
This is free and open source software.
Key Features
- Efficient execution of Transformer models on CPUs and GPUs.
- Support for encoder-decoder, decoder-only and encoder-only neural network architectures.
- Compatible with models including BERT, T5, Whisper, Llama, Mistral, Gemma and Qwen.
- Model conversion tools for OpenNMT, Marian, Fairseq, OPUS-MT and Hugging Face Transformers.
- Weight quantisation using reduced-precision numerical formats to lower memory requirements.
- Support for FP16, BF16, INT16, INT8 and selected INT4 quantisation methods.
- Optimisations including layer fusion, padding removal, batch reordering and in-place operations.
- Automatic selection of optimised CPU computation backends.
- Parallel and asynchronous inference across CPU cores and supported GPUs.
- Tensor parallelism for distributing large models across multiple GPUs.
- Dynamic memory allocation and caching for varying inference workloads.
- Batch translation and text generation APIs.
- Configurable beam search, sampling and other decoding strategies.
- Interactive decoding with partial sequence completion and alternative predictions.
- Support for automatic speech recognition using compatible Whisper models.
- C++ and Python interfaces for integration into other applications.
- Cross-platform support – runs on Linux, macOS, and Windows.
Website: github.com/OpenNMT/CTranslate2
Support:
Developer: OpenNMT authors and SYSTRAN contributors
License: MIT License
CTranslate2 is written in C++. Learn C++ with our recommended free books and free tutorials. Learn Python with our recommended free books and free tutorials.
Related Software
| C++ Natural Language Processing Tools | |
|---|---|
| text2vec | Framework with API for text analysis and natural language processing |
| Moses | Statistical machine translation system |
| TiMBL | Tilburg Memory-Based Learner |
| MITIE | MIT Information Extraction |
| MeTA | Modern C++ data sciences toolkit |
| Colibri Core | Efficient n-gram & skipgram modelling on text corpora |
| CRF++ | Yet Another CRF toolkit |
| BLLIP Parser | Statistical natural language parser |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |
Please read our Comment Policy before commenting.