Natural Language Processing

tokenizers – fast and consistent text tokenisation in R

tokenizers is a collection of functions for converting natural language text into smaller units suitable for computational analysis. These units, known as tokens, can represent words, sentences, characters, paragraphs or combinations of neighbouring words.

The package provides a consistent interface for different tokenisation strategies, making it straightforward to select an approach according to the requirements of a particular text-analysis task. Researchers might use individual word tokens for frequency analysis, n-grams for investigating recurring phrases or sentence tokens for sentiment analysis.

tokenizers is designed to fit into larger analytical workflows. Its functions accept text collections and return tokenised results that can be processed by other packages for text mining, statistical analysis or machine learning.

This is free and open source software.

Key Features

  • Converts text into individual word tokens.
  • Splits documents into sentences, paragraphs and lines.
  • Provides character-level tokenisation and character shingles.
  • Creates contiguous word n-grams for identifying recurring expressions.
  • Supports skip n-grams containing nonadjacent terms.
  • Includes Penn Treebank tokenisation.
  • Offers tokenisation using configurable regular expressions.
  • Supports word stemming through a dedicated tokenisation function.
  • Counts words, characters and sentences in textual data.
  • Divides longer texts into smaller documents containing specified numbers of words.
  • Uses Unicode-aware string processing for handling UTF-8 text.
  • Provides consistent output structures across different tokenisation methods.
  • Works with tidytext and other R text-mining packages.

Website: github.com/ropensci/tokenizers
Support:
Developer: Lincoln Mullen
License: MIT License

tokenizers is written in R. Learn R with our recommended free books and free tutorials.


Related Software

R Natural Language Processing Tools
tidytextText mining using dplyr, ggplot2, and other tidy tools
quantedaR package for Quantitative Analysis of Textual Data
text2vecFramework with API for text analysis and natural language processing
wordcloudCreate attractive word clouds
tmText Mining Infrastructure in R
srtringiFast and portable character string processing in R
StringrString manipulation in R
UDPipeTokenization, Tagging, Lemmatization and Dependency Parsing
tokenizersConvert natural language text into tokens
spacyrR wrapper around the Python spaCy package
Word VectorsBuild and explore embedding models
syuzhetExtraction of sentiment and sentiment-based plot arcs from text
textTinyRText processing for small or big data
sentimentrDictionary based sentiment analysis
textcleanCollection of tools to clean and normalize text
TALLExplore, model, and visualize textual data
corpustoolsVarious tools for analyzing text corpora
topicmodelsInterface to LDA and CTM models
textAnalyzing natural language with transformers-based large language models
RTextToolsAutomatic text classification via supervised learning

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted