tokenizers is a collection of functions for converting natural language text into smaller units suitable for computational analysis. These units, known as tokens, can represent words, sentences, characters, paragraphs or combinations of neighbouring words.
The package provides a consistent interface for different tokenisation strategies, making it straightforward to select an approach according to the requirements of a particular text-analysis task. Researchers might use individual word tokens for frequency analysis, n-grams for investigating recurring phrases or sentence tokens for sentiment analysis.
tokenizers is designed to fit into larger analytical workflows. Its functions accept text collections and return tokenised results that can be processed by other packages for text mining, statistical analysis or machine learning.
This is free and open source software.
Key Features
- Converts text into individual word tokens.
- Splits documents into sentences, paragraphs and lines.
- Provides character-level tokenisation and character shingles.
- Creates contiguous word n-grams for identifying recurring expressions.
- Supports skip n-grams containing nonadjacent terms.
- Includes Penn Treebank tokenisation.
- Offers tokenisation using configurable regular expressions.
- Supports word stemming through a dedicated tokenisation function.
- Counts words, characters and sentences in textual data.
- Divides longer texts into smaller documents containing specified numbers of words.
- Uses Unicode-aware string processing for handling UTF-8 text.
- Provides consistent output structures across different tokenisation methods.
- Works with tidytext and other R text-mining packages.
Website: github.com/ropensci/tokenizers
Support:
Developer: Lincoln Mullen
License: MIT License
tokenizers is written in R. Learn R with our recommended free books and free tutorials.
Related Software
| R Natural Language Processing Tools | |
|---|---|
| tidytext | Text mining using dplyr, ggplot2, and other tidy tools |
| quanteda | R package for Quantitative Analysis of Textual Data |
| text2vec | Framework with API for text analysis and natural language processing |
| wordcloud | Create attractive word clouds |
| tm | Text Mining Infrastructure in R |
| srtringi | Fast and portable character string processing in R |
| Stringr | String manipulation in R |
| UDPipe | Tokenization, Tagging, Lemmatization and Dependency Parsing |
| tokenizers | Convert natural language text into tokens |
| spacyr | R wrapper around the Python spaCy package |
| Word Vectors | Build and explore embedding models |
| syuzhet | Extraction of sentiment and sentiment-based plot arcs from text |
| textTinyR | Text processing for small or big data |
| sentimentr | Dictionary based sentiment analysis |
| textclean | Collection of tools to clean and normalize text |
| TALL | Explore, model, and visualize textual data |
| corpustools | Various tools for analyzing text corpora |
| topicmodels | Interface to LDA and CTM models |
| text | Analyzing natural language with transformers-based large language models |
| RTextTools | Automatic text classification via supervised learning |
Read our verdict in the software roundup.
LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |


Please read our Comment Policy before commenting.