Quantcast File System
Quantcast File System (QFS) is a high-performance, open source,
fault-tolerant, distributed file system developed to support MapReduce
processing, or other applications reading and writing large files
sequentially. It is an alternative to HDFS for large-scale batch
data processing.
QFS consists of 3 components:
- Metaserver - a central process metadata server that manages
the directory structure and maps of files to physical storage.
- Chunk Server - the distributed component of the distributed
file system. A chunk server runs on each machine that will
host data, manages I/O to its hard drives, and monitors its activity
and capacity.
- Client Library - a library that provides the file system
API to allow applications to interface with QFS. It makes
requests of the metaserver to identify which chunk servers hold (or
will hold) its data, then interacts with the chunk servers directly to
read and write.
QFS is written in C++, operates within a fixed memory footprint, and uses direct I/O.
Features include:
- Incremental Scalability
- Balancing
- Rebalancing
- Fault Tolerance
- Fine-tunable Replication, Striping, Recovery Mode
- Re-replication
- Data Integrity
- Reed-Solomon (RS) error correction
- Client Side Metadata Caching
- File Writes
- Leases
- Versioning
- Client Side Fail-over
- Direct I/O
- Manages its own memory within a fixed footprint
- Language Support
- Tools
- FUSE support on Linux
- Unix style permissions support
Return
to File Systems Home Page
Last Updated Friday, April 12 2013 @ 09:42 PM EDT
| 
