Hadoop Distributed File System (HDFS) is the storage component of Apache Hadoop. It is designed to hold very large datasets across clusters of computers and provide high-throughput access for applications that process data in parallel.
HDFS separates filesystem metadata from the actual file contents. NameNodes manage directories, file attributes and block locations, while DataNodes store the underlying data blocks. Clients obtain block information from a NameNode and communicate directly with DataNodes to transfer file contents.
Its architecture favours large files and streaming workloads rather than low-latency access to small files. HDFS is particularly suited to batch processing, analytics and workloads built around the Hadoop ecosystem.
This is free and open source software.
Key Features
- Stores large files as blocks distributed across multiple DataNodes.
- Supports block replication with configurable replication factors and rack-aware placement.
- Offers erasure coding to reduce storage overhead for suitable datasets.
- Provides NameNode high availability using active and standby services.
- Supports filesystem federation to distribute namespace management across multiple NameNodes.
- Includes point-in-time directory snapshots and configurable storage quotas.
- Offers transparent encryption for protected filesystem zones.
- Provides checksums to detect data corruption during storage and transfer.
- Includes a balancer for redistributing blocks across storage nodes.
- Supports WebHDFS, filesystem shell tools and programmatic access through Hadoop interfaces.
- Provides access controls using filesystem permissions and access control lists.
Website: hadoop.apache.org
Support:
Developer: The Apache Software Foundation
License: Apache License 2.0
HDFS is written in Java. Learn Java with our recommended free books and free tutorials.
Related Software
| File Systems | |
|---|---|
| HDFS | Distributed file system providing high-throughput access |
| SeaweedFS | Simple and highly scalable distributed file system |
| Lustre | File system for computer clusters |
| CephFS | Unified, distributed storage system |
| Alluxio | Virtual distributed file system |
| GlusterFS | Scale-out NAS file system |
| JuiceFS | Distributed POSIX file system |
| XtreemFS | Object-based, distributed file system for wide area networks |
| MooseFS | POSIX-compliant distributed file system |
| Quantcast File System | High-performance, fault-tolerant, distributed file system |
| OrangeFS | Multi-server scalable parallel file system |
| LeilFS | Distributed POSIX file system |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |
Please read our Comment Policy before commenting.