The Hadoop Distributed File System (HDFS) is a distributed, scalable, and portable file system written in Java for the Hadoop framework. It provides high-throughput access to application data, and similar functionality to that provided by the Google File System.
HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS is suitable for applications that have large data sets. HDFS relaxes a few POSIX requirements to enable streaming access to file system data.
Each node in a Hadoop instance typically has a single namenode; a cluster of datanodes form the HDFS cluster. The situation is typical because each node does not require a datanode to be present. Each datanode serves up blocks of data over the network using a block protocol specific to HDFS.
HDFS is designed to scale to tens of petabytes of storage and runs on top of the filesystems of the underlying operating systems. It is a sub-project of the Apache Hadoop project.
- Supports very large files.
- Master/slave architecture.
- Simple Coherency Model.
- Data access via MapReduce streaming.
- Easily portable from one platform to another.
- Supports a traditional hierarchical file organization.
- Designed to reliably store very large files across machines in a large cluster. It stores each file as a sequence of blocks; all blocks in a file except the last block are the same size.
- Blocks of a file are replicated for fault tolerance across multiple hosts, avoiding the need for RAID storage.
- Persistence of File System Metadata.
- HDFS communication protocols are layered on top of the TCP/IP protocol.
- Compatible with data rebalancing schemes.
- Checksum checking on the contents of HDFS files.
Support: Users Guide
Developer: The Apache Software Foundation
License: Apache License 2.0
HDFS is written in Java. Learn Java with our recommended free books and free tutorials.
Return to File Systems Home Page
|The largest compilation of the best free and open source software in the universe. Each article is supplied with a legendary ratings chart helping you to make informed decisions.|
|Hundreds of in-depth reviews offering our unbiased and expert opinion on software. We offer helpful and impartial information.|
|Replace proprietary software with open source alternatives: Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle, Atlassian, Corel, Cisco, Intuit, and SAS.|
|Machine Learning explores practical applications of machine learning and deep learning from a Linux perspective. This is a new series.|
|New to Linux? Read our Linux for Starters series. We start right at the basics and teach you everything you need to know to get started with Linux.|
|Essential Linux system tools focuses on small, indispensable utilities, useful for system administrators as well as regular users.|
|Linux utilities to maximise your productivity. Small, indispensable tools, useful for anyone running a Linux machine.|
|Surveys popular streaming services from a Linux perspective: Amazon Music Unlimited, Myuzi, Spotify, Deezer, Tidal.|
|Saving Money with Linux looks at how you can reduce your energy bills running Linux.|
|Home computers became commonplace in the 1980s. Emulate home computers including the Commodore 64, Amiga, Atari ST, ZX81, Amstrad CPC, and ZX Spectrum.|
|Now and Then examines how promising open source software fared over the years. It can be a bumpy ride.|
|Linux at Home looks at a range of home activities where Linux can play its part, making the most of our time at home, keeping active and engaged.|
|Linux Candy reveals the lighter side of Linux. Have some fun and escape from the daily drudgery.|
|Getting Started with Docker helps you master Docker, a set of platform as a service products that delivers software in packages called containers.|
|Best Free Android Apps. We showcase free Android apps that are definitely worth downloading. There's a strict eligibility criteria for inclusion in this series.|
|These best free books accelerate your learning of every programming language. Learn a new language today!|
|These free tutorials offer the perfect tonic to our free programming books series.|
|Linux Around The World showcases usergroups that are relevant to Linux enthusiasts.|
|Stars and Stripes is an occasional series looking at the impact of Linux in the USA.|