LinuxLinks.com
Newbies What Next ? News Forums Calendar

Search





News Sections
Home
General News (3973/0)
Reviews (628/0)
Press Releases (464/0)
Distributions (187/0)
Software (857/0)
Hardware (522/0)
Security (192/0)
Tutorials (338/0)
Off Topic (181/0)


User Functions
Username:

Password:

Don't have an account yet? Sign up as a New User


Events
There are no upcoming events




Nutch

Apache Nutch

Apache Nutch is an open source web-search software project written in Java. Find Web page hyperlinks in an automated manner, reduce lots of maintenance work, for example checking broken links, and create a copy of all the visited pages for searching over.

It has been used in a variety of different applications such as vertical search engines, archival web search, search engines that incorporate novel metadata, and others.

Apache Nutch supports Solr out-the-box, simplifying Nutch-Solr integration. It also removes the legacy dependence upon both Apache Tomcat for running the old Nutch Web Application and upon Apache Lucene for indexing.

Apache Nutch uses the PDFBox API in its parse-tika plugin for extracting textual content and metadata from encrypted PDF files. Nutch can run on a single machine, but performance is improved in Hadoop clusters.

 Nutch 1.7

Price
Free to download

Size
2.6MB
License

Apache License Version 2.0

Developer
Apache Software Foundation

Website
nutch.apache.org

System Requirements
Java 6

Support Sites:
FAQ, Wiki, Mailing Lists, Tutorial

Selected Reviews:

Features include:

  • Crawler
  • Fetching and parsing are done separately by default, reducing the risk of an error corrupting the fetch parse stage of a crawl
  • MapReduce
  • Distributed filesystem (via Hadoop)
  • Link-graph database
  • NTLM authentication
  • Flexible, easily extensible plugin infrastructure
  • Parsing support handled by Apache Tika
  • The number of plugins for processing various document types being shipped with Nutch has been refined.
    Supports:
    • Plain Text (plugin: tika)
    • HTML/XHTML+XML (parse-html/tika)
    • XML (parse-Tika/feed) uses XPath and namespaces to do the mapping between XML elements and index fields
    • JavaScript (parse-js)
    • OpenOfice.org ODF (parse-tika) parses Open Office and Star Office documents
    • Microsoft Power Point, the .ppt file (parse-tika)
    • Microsoft Word, the .doc file (parse-tika)
    • Adobe PDF (parse-tika)
    • RSS (parse-feed/tika)
    • RTF (parse-tika)
    • MP3 (parse-tika) The mp3 itself contains the ID3v1 or ID3v2 tags which contain metadata song information like title, artist, album, comments, etc. The useful information needed to search mp3s
    • ZIP (parse-zip) This seems to expand the zip of plain text files and return the concatenated text

Return to Search Engines for Big Data Home Page

Bookmark and Share


Last Updated Tuesday, October 08 2013 @ 04:07 PM EDT


We have written a range of guides highlighting excellent free books for popular programming languages. Check out the following guides: C, C++, C#, Java, JavaScript, CoffeeScript, HTML, Python, Ruby, Perl, Haskell, PHP, Lisp, R, Prolog, Scala, Scheme, Forth, and SQL.


Group Tests
All Group Tests

Top Free Software
6 Office Suites
7 Document Processors
6 Lean Desktops
6 Desktop Search
9 Project Management
9 Groupware Apps
14 File Managers
10 Databases
21 Backup Tools
21 DVD Tools
21 Window Managers
21 Productivity Tools
9 Terminal Emulators
21 Financial Tools
21 Text Editors
21 Video Emulators
21 Home Emulators
42 Graphics Apps
6 CAD Apps
42 Scientific Apps
10 Web Browsers
42 Email Apps
12 Instant Messaging
10 IRC Clients
7 Twitter Clients
12 News Aggregators
11 VoIP Apps
11 Remote Display Apps
42 Best Games
42 More Games
21 More Games
21 Notable Games (1)
21 Notable Games (2)
21 Notable Games (3)
8 ASCII Games
9 Educational Games
42 Audio Apps
42 Video Apps
6 Screencasting Apps
80 Security Apps
9 System Monitoring
6 Family History Apps
11 PDF Tools
6 Music Servers
6 Collection Managers
7 Calculator Apps
8 Geometry Apps
Free Console Apps
14 Multimedia
Programming
8 Compilers
9 IDEs
9 Debuggers
7 Revision Control Apps
6 Doc Generators
'Free' Proprietary
21 Closed-Source Apps
Top Commercial Apps
42 Games
Free Web Software
21 Web CMS
14 Wiki Engines
8 Blog Apps
6 eCommerce Apps
5 Human Resource Apps
10 ERP
10 CRM
6 Data Warehouse Apps
8 Business Intelligence
6 Point-of-Sale

All Group Tests

Other Articles
Migrating from Windows
Back up your data
Distribution Guide
Distro Portal Pages
20 Free Linux Books
Running Linux Under Windows


Older Stories
Sunday 02/09
  • 5 Highly Promising Cross-Platform IDEs (0)
  • Play the Best Linux Puzzle Games (0)
  • Essential LaTeX Tools (0)
  • Unleashing the Best Open Source Social Networking Software (0)
  • Media Streaming with Top UPnP Servers (0)
  • Highlighting the Top Open Source Pandora Radio Clients (0)

  • Tuesday 12/24
  • Spotlighting the Top Open Source Crafting Tools (0)

  • Sunday 12/15
  • Uncovering the Best Open Source Google Analytics Alternatives (0)
  • Top Email Encryption Tools (0)

  • Saturday 11/30
  • Small Image Tools that Pack a Real Punch (0)


  • Vote

    What do you find MOST attractive about Open Source software?

    Amount of customization
    Security
    Freedom provided
    Speed of development
    Quality
    Multiple versions
    Cost
    Potential to contribute
    Ability to modify code
    Results
    4689 votes | 2 comments

    Built with GeekLog and phpBB
    Comments to the webmaster are welcome
    Copyright 2009 LinuxLinks.com - All rights reserved