3FS (Fire-Flyer File System) is a distributed file system for demanding AI training and inference workloads. It combines high-speed network interconnects with modern NVMe storage to provide shared access to data across large computing clusters.
The system uses a disaggregated architecture in which storage resources are distributed across dedicated nodes. Applications can access a shared namespace without needing to know which storage device or node holds a particular file.
3FS is designed for workloads that need both high aggregate throughput and efficient random access. These include preparing large datasets, supplying training samples to accelerators, saving model checkpoints and retaining intermediate data produced during inference.
Its architecture separates metadata management from storage services and uses strong consistency mechanisms to simplify coordination between applications operating on the same datasets.
This is free and open source software.
Key Features
- Aggregates storage capacity and bandwidth across multiple storage nodes.
- Uses NVMe SSDs and RDMA-capable networks for high-throughput data access.
- Provides a disaggregated storage architecture that separates computation from persistent data.
- Implements Chain Replication with Apportioned Queries (CRAQ) for strongly consistent operations.
- Uses stateless metadata services backed by a transactional key-value store.
- Supports FoundationDB for persistent metadata management.
- Provides a hierarchical file namespace for organising shared datasets.
- Offers a FUSE interface for accessing files through conventional filesystem operations.
- Provides the USRBIO interface for applications requiring direct storage access.
- Supports high-throughput parallel checkpointing of large machine learning models.
- Enables random access to training samples across distributed compute nodes.
- Supports storage-backed key-value caching for inference workloads.
- Provides tools for evaluating storage throughput using fio.
- Supports large-scale data preparation pipelines and intermediate datasets.
Website: github.com/deepseek-ai/3FS
Support:
Developer: DeepSeek
License: MIT License
3FS is written in C++. Learn C++ with our recommended free books and free tutorials.
Related Software
| File Systems | |
|---|---|
| HDFS | Distributed file system providing high-throughput access |
| SeaweedFS | Simple and highly scalable distributed file system |
| Lustre | File system for computer clusters |
| CephFS | Unified, distributed storage system |
| Alluxio | Virtual distributed file system |
| GlusterFS | Scale-out NAS file system |
| JuiceFS | Distributed POSIX file system |
| XtreemFS | Object-based, distributed file system for wide area networks |
| MooseFS | POSIX-compliant distributed file system |
| Quantcast File System | High-performance, fault-tolerant, distributed file system |
| OrangeFS | Multi-server scalable parallel file system |
| LeilFS | Distributed POSIX file system |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |
Please read our Comment Policy before commenting.