Natural Language Processing

Apache Lucene – full-text search engine library

Apache Lucene is an information retrieval library for applications that need to index and search text and other structured information. It provides the underlying search technology used by numerous search servers and document retrieval systems.

Lucene manages inverted indexes and provides APIs for adding documents, searching indexed fields and ranking results according to their relevance. Unlike a complete search server, it is intended to be integrated into applications rather than operated as a standalone service.

The library supports a wide range of search operations, including Boolean expressions, phrase and proximity queries, filtering and relevance-based retrieval. It also provides facilities for faceted navigation, spelling suggestions and highlighting matching terms in search results.

Lucene can index numerical and vector data alongside textual information. This enables applications to combine conventional document retrieval with nearest-neighbour search over high-dimensional vectors.

Developers can customise analysis, ranking and storage behaviour to suit different document collections and retrieval requirements.

This is free and open source software.

Key Features

  • High-performance incremental document indexing.
  • Inverted indexes for efficient full-text retrieval.
  • Fielded searches across structured document information.
  • Boolean, phrase, proximity, wildcard and range queries.
  • Relevance ranking with BM25 and other scoring models.
  • Nearest-neighbour search over high-dimensional vectors.
  • Support for combining lexical and vector-based retrieval.
  • Faceted navigation, result grouping and highlighting.
  • Text analysis with configurable tokenisers and filters.
  • Stemming, synonym handling and language-aware processing.
  • Spelling correction and query suggestions.
  • Concurrent indexing and searching with near real-time readers.
  • Configurable index codecs and storage structures.
  • Search across multiple indexes and merge the results.
  • Cross-platform support – runs on Linux, macOS, and Windows.

Website: lucene.apache.org/core
Support:
Developer: Apache Software Foundation
License: Apache License 2.0

Apache Lucene is written in Java. Learn Java with our recommended free books and free tutorials.


Related Software

Search Engines for Big Data
ElasticSearchFlexible and powerful distributed RESTful search engine and analytics
SolrSearch engine server that uses Lucene
LuceneSearch engine library
QuickwitCloud-native search engine
MeiliSearchEasy to use and deploy search engine
OpenSearchEnterprise-grade search and observability suite
SphinxSearch engine designed with indexing database content in mind
XapianProbabilistic information retrieval library
TypesenseFast, typo-tolerant search engine
Manticore SearchEasy to use fast database for search
RediSearchQuery and indexing engine for Redis
ParadeDBPostgreSQL extension adding search and analytical capabilities to Postgres
VespaAI search platform
MroongaMySQL storage engine based on the Groonga search engine
PGroongaAdds a Groonga-powered index access method
GroongaFulltext search engine and column store

Read our verdict in the software roundup.

Natural Language Processing
PyTorch-TransformersLibrary of state-of-the-art pre-trained models
Natural Language ToolkitSuite of open source Python modules, data sets and tutorials
Stanford CoreNLPExtensible annotation-based NLP pipeline
spaCyIndustrial strength natural language processing
scikit-learnMachine learning library for Python
GensimPython-based vector space modeling and topic modeling toolkit
flairSimple framework for state-of-the-art NLP
Apache OpenNLPMachine learning based toolkit
DL4JDeploy and train deep learning models
Apache LuceneFull-featured information retrieval software library
UIMAImplementation of the UIMA specification
tidytextText mining using dplyr, ggplot2, and other tidy tools
text2vecFramework with API for text analysis and NLP
quantedaR package for Quantitative Analysis of Textual Data
MosesStatistical machine translation system

Read our verdict in the software roundup.

Java Natural Language Processing Tools
CoreNLPAnnotation-based NLP pipeline that provides core natural language analysis
OpenNLPMachine learning based toolkit
DL4JDeploy and train deep learning models
LuceneHigh-performance, full-featured information retrieval software library
UIMAOpen source implementation of the UIMA specification
TikaContent analysis toolkit
MALLETStatistical natural language processing, document classification and more
GATEFull-lifecycle solution for a broad range of NLP tasks
ReVerb Automatically identifies and extracts binary relationships from sentences
NLP4JNLP framework for JVM languages
CogComp-NLPState-of-the-art Natural Language Processing (NLP) tools

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted