Spark NLP is a natural language processing library built on Apache Spark. It provides tools for analysing text and other forms of unstructured information using distributed processing pipelines, making it suitable for applications that need to process large datasets.
The library combines traditional linguistic analysis with modern deep learning techniques. It supports a broad collection of pretrained models that can be incorporated into workflows without requiring every model to be trained from scratch.
Spark NLP is used for tasks ranging from identifying entities in documents to classifying text, translating languages and generating natural language. Its integration with Apache Spark allows processing to scale beyond a single machine, while interfaces for Python, Scala and Java make it accessible to different development environments.
This is free and open source software.
Key Features
- Supports tokenisation, sentence detection, lemmatisation and part-of-speech tagging.
- Provides named entity recognition and entity extraction.
- Offers sentiment analysis and text classification.
- Supports dependency parsing and linguistic annotation.
- Generates word and sentence embeddings for downstream analysis.
- Includes machine translation, text summarisation and question answering.
- Supports text generation and zero-shot classification.
- Provides automatic speech recognition and image-related processing tasks.
- Offers a large collection of pretrained pipelines and models across numerous languages.
- Supports transformer architectures including BERT, RoBERTa, T5 and other model families.
- Can import models from TensorFlow, ONNX, OpenVINO and GGUF-compatible frameworks.
- Integrates with Apache Spark ML pipelines for distributed processing.
- Provides interfaces for Python, Scala, Java and other JVM languages.
- Supports CPU processing and GPU acceleration in compatible environments.
- Cross-platform support – runs on Linux, macOS, and Windows.
Website: github.com/JohnSnowLabs/spark-nlp
Support:
Developer: John Snow Labs
License: Apache License 2.0
Spark NLP is written in Scala. Learn Scala with our recommended free books and free tutorials.
Related Software
| Java Natural Language Processing Tools | |
|---|---|
| CoreNLP | Annotation-based NLP pipeline that provides core natural language analysis |
| OpenNLP | Machine learning based toolkit |
| DL4J | Deploy and train deep learning models |
| Lucene | High-performance, full-featured information retrieval software library |
| UIMA | Open source implementation of the UIMA specification |
| Tika | Content analysis toolkit |
| MALLET | Statistical natural language processing, document classification and more |
| GATE | Full-lifecycle solution for a broad range of NLP tasks |
| ReVerb | Automatically identifies and extracts binary relationships from sentences |
| NLP4J | NLP framework for JVM languages |
| CogComp-NLP | State-of-the-art Natural Language Processing (NLP) tools |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |
Please read our Comment Policy before commenting.