text2vec is an R package which provides an efficient framework with a concise API for text analysis and natural language processing (NLP). Fast vectorization, topic modeling, distances and GloVe word embeddings in R.
This package is efficient because it is carefully written in C++, which also means that text2vec is memory friendly. Some parts (such as GloVe) are fully parallelized using the RcppParallel package.
text2vec is inspired by gensim. The software is released under an open source license.
Key Features
- Fast text vectorization (creation of document-term matrices) on arbitrary n-grams, using vocabulary or feature hashing.
- GloVe word embeddings.
- Topic modelling with:
- Latent Dirichlet Allocation – decomposes document-term matrix into two low-rank matrices – document-topic distribution and topic-word distribution. It’s a more complex non-linear generative model.
- Latent Semantic Analysis – oldest among topic modeling techniques. It also decomposes Document-Term matrix into a product of 2 low rank matrices.
- Provides 2 set of functions for measuring various distances/similarity in a unified way. All methods are written with special attention to computational performance and memory efficiency. Functions for fast calculation of similarity between documents (specifically similarities and distances between matrices):
- Cosine distance – classical approach from computational linguistics is to measure similarity based on the content overlap between documents.
- Jaccard distance – a simple but intuitive measure of similarity between two sets.
- Relaxed Word Mover’s Distance.
- Euclidean – with text2vec it can only be computed on dense matrices.
- Streaming API means that users do not have to load all the data into RAM:
- Vectorization.
- I/O handling.
- Easy parallel processing – provides convenient functions for easy parallel processing of text.
- Models – provides unified interface for models, which is inspired by scikit-learn interface. Models in text2vec are mostly transformers and decompositions.
Website: text2vec.org
Support: GitHub Code Repository
Developer: Dmitry Selivanov and contributors
License: GNU General Public License v2.0
text2vec is written in R. Learn R with our recommended free books and free tutorials.
Related Software
| Natural Language Processing | |
|---|---|
| PyTorch-Transformers | Library of state-of-the-art pre-trained models |
| Natural Language Toolkit | Suite of open source Python modules, data sets and tutorials |
| Stanford CoreNLP | Extensible annotation-based NLP pipeline |
| spaCy | Industrial strength natural language processing |
| scikit-learn | Machine learning library for Python |
| Gensim | Python-based vector space modeling and topic modeling toolkit |
| flair | Simple framework for state-of-the-art NLP |
| Apache OpenNLP | Machine learning based toolkit |
| DL4J | Deploy and train deep learning models |
| Apache Lucene | Full-featured information retrieval software library |
| UIMA | Implementation of the UIMA specification |
| tidytext | Text mining using dplyr, ggplot2, and other tidy tools |
| text2vec | Framework with API for text analysis and NLP |
| quanteda | R package for Quantitative Analysis of Textual Data |
| Moses | Statistical machine translation system |
Read our verdict in the software roundup.
| R Natural Language Processing Tools | |
|---|---|
| tidytext | Text mining using dplyr, ggplot2, and other tidy tools |
| quanteda | R package for Quantitative Analysis of Textual Data |
| text2vec | Framework with API for text analysis and natural language processing |
| wordcloud | Create attractive word clouds |
| tm | Text Mining Infrastructure in R |
| srtringi | Fast and portable character string processing in R |
| Stringr | String manipulation in R |
| UDPipe | Tokenization, Tagging, Lemmatization and Dependency Parsing |
| tokenizers | Convert natural language text into tokens |
| spacyr | R wrapper around the Python spaCy package |
| Word Vectors | Build and explore embedding models |
| syuzhet | Extraction of sentiment and sentiment-based plot arcs from text |
| textTinyR | Text processing for small or big data |
| sentimentr | Dictionary based sentiment analysis |
| textclean | Collection of tools to clean and normalize text |
| corpustools | Various tools for analyzing text corpora |
| topicmodels | Interface to LDA and CTM models |
| text | Analyzing natural language with transformers-based large language models |
| RTextTools | Automatic text classification via supervised learning |
Read our verdict in the software roundup.
| C++ Natural Language Processing Tools | |
|---|---|
| text2vec | Framework with API for text analysis and natural language processing |
| Moses | Statistical machine translation system |
| TiMBL | Tilburg Memory-Based Learner |
| MITIE | MIT Information Extraction |
| MeTA | Modern C++ data sciences toolkit |
| Colibri Core | Efficient n-gram & skipgram modelling on text corpora |
| CRF++ | Yet Another CRF toolkit |
| BLLIP Parser | Statistical natural language parser |
Read our verdict in the software roundup.
 Explore our comprehensive directory of recommended free and open source software. Our carefully curated collection spans every major software category.This directory is part of our ongoing series of informative articles for Linux enthusiasts. It features hundreds of detailed reviews, along with open source alternatives to proprietary solutions from major corporations such as Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. You’ll also find interesting projects to try, hardware coverage, free programming books and tutorials, and much more. Discovered a useful open source Linux program that we haven’t covered yet? Let us know by completing this form. |