Nutch – highly extensible, highly scalable Web crawler

Last Updated on July 25, 2023

Apache Nutch is an open source web-search software project written in Java. Find Web page hyperlinks in an automated manner, reduce lots of maintenance work, for example checking broken links, and create a copy of all the visited pages for searching over.

It has been used in a variety of different applications such as vertical search engines, archival web search, search engines that incorporate novel metadata, and others.

Apache Nutch supports Solr out-the-box, simplifying Nutch-Solr integration. It also removes the legacy dependence upon both Apache Tomcat for running the old Nutch Web Application and upon Apache Lucene for indexing.

Apache Nutch uses the PDFBox API in its parse-tika plugin for extracting textual content and metadata from encrypted PDF files. Nutch can run on a single machine, but performance is improved in Hadoop clusters.

Features include:

• Crawler.
• Fetching and parsing are done separately by default, reducing the risk of an error corrupting the fetch parse stage of a crawl.
• MapReduce.
• Distributed filesystem (via Hadoop).
• Link-graph database.
• NTLM authentication.
• Flexible, easily extensible plugin infrastructure.
• Parsing support handled by Apache Tika.
• The number of plugins for processing various document types being shipped with Nutch has been refined.
•  Supports:
  • Plain Text (plugin: tika).
• • HTML/XHTML+XML (parse-html/tika).
• • XML (parse-Tika/feed) uses XPath and namespaces to do the mapping between XML elements and index fields.
  • JavaScript (parse-js).
  • OpenOfice.org ODF (parse-tika) parses Open Office and Star Office documents.
• • Microsoft Power Point, the .ppt file (parse-tika).
• • Microsoft Word, the .doc file (parse-tika).
• • Adobe PDF (parse-tika).
• • RSS (parse-feed/tika).
• • RTF (parse-tika).
• • MP3 (parse-tika) The mp3 itself contains the ID3v1 or ID3v2 tags which contain metadata song information like title, artist, album, comments, etc. The useful information needed to search mp3s.
• • ZIP (parse-zip) This seems to expand the zip of plain text files and return the concatenated text.

Website: nutch.apache.org
Support: Wiki
Developer: Apache Software Foundation
License: Apache License Version 2.0

Apache Nutch is written in Java. Learn Java with our recommended free books and free tutorials.

Return to Search Engines for Big Data

Explore Linux
	Best Free and Open Source Software - A huge collection of recommended free and open source software covering every major category.
	In-depth Linux Software Reviews - Detailed assessments of Linux software covering features, usability and overall quality.
	The Big List of Active Linux Distros - An extensive collection of actively developed Linux distributions.
	Open Source Alternatives - Alternatives to software and services from Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle and others.
	Essential Linux System Tools - Indispensable utilities for monitoring, maintaining and managing Linux systems.
	Linux Productivity Tools - Useful utilities for improving productivity and streamlining everyday tasks.
	Alternatives to Popular CLI Tools - Modern alternatives to familiar command-line utilities supplied with Linux systems.
	Awesome Free Linux Game Tools - Tools, utilities and companion software that improve gaming on Linux.
	Machine Learning on Linux - Machine learning and deep learning software, self-hosted applications and practical tools.
	Saving Money with Linux - Practical ways to reduce electricity consumption and lower computing costs with Linux.
	Linux Candy - Fun, entertaining and sometimes delightfully quirky software from the lighter side of Linux.