# Fire-Flyer File System (3FS)

> Source: https://aiwiki.ai/wiki/fire_flyer_file_system
> Updated: 2026-09-23
> Fact-checked: 2026-09-23
> Categories: AI Infrastructure, Chinese AI, Open Source AI
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Fire-Flyer File System (3FS)." aiwiki.ai, 23 Sept 2026. https://aiwiki.ai/wiki/fire_flyer_file_system
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

The **Fire-Flyer File System (3FS)** is a distributed file system built by [DeepSeek](https://aiwiki.ai/wiki/deepseek) and the Chinese quantitative fund [High-Flyer](https://aiwiki.ai/wiki/high_flyer) for AI training and inference workloads. It pools the SSDs of many storage servers behind an RDMA network (InfiniBand or RoCE), keeps file metadata in the transactional key-value store FoundationDB, and replicates data with Chain Replication with Apportioned Queries (CRAQ) so that every read sees strongly consistent data.[1][2] The name comes from the Fire-Flyer GPU clusters it was written for; High-Flyer shortened it to "3FS" because the three consecutive F's were awkward to say.[8] 3FS was introduced on High-Flyer's technical blog, described in detail in the Fire-Flyer AI-HPC paper presented at SC24, and released by DeepSeek as open source under the MIT License on 28 February 2025, on day five, the last of the five daily repository releases in its "Open Source Week", together with a companion data-processing framework called smallpond.[3][4][5] DeepSeek says 3FS handles training data preprocessing, dataset loading, checkpoint saving and reloading, embedding vector search and KV cache lookups for inference, and DeepSeek's 2026 papers describe it as the image store behind the company's [DSec](https://aiwiki.ai/wiki/dsec) sandbox platform for agentic reinforcement learning.[4][13]

## History

### High-Flyer and the Fire-Flyer clusters

3FS grew out of the in-house computing effort of High-Flyer (幻方), a Hangzhou quantitative fund whose staff built Fire-Flyer 2 together with DeepSeek-AI.[3][7] High-Flyer's corporate timeline says it registered a separate AI research company in 2019 and built its first cluster, Fire-Flyer 1 (萤火一号), in-house with 500 graphics cards and a 200 Gbps interconnect. The 2020 entry describes Fire-Flyer 1 as a total investment of nearly 200 million yuan carrying 1,100 accelerator cards, put into service that year. In 2021 High-Flyer invested one billion yuan in Fire-Flyer 2 (萤火二号), retired Fire-Flyer 1 after about a year and a half, and listed a self-developed distributed parallel file system, 3FS, among the pieces of Fire-Flyer 2's first phase alongside 200 Gbps networking, the hfreduce communication scheme and the hfai.nn operator library.[7]

A High-Flyer technical blog post introduced 3FS as a key part of the storage service in Fire-Flyer 2 after compute and storage were separated. It described a system built almost entirely for one job: compute nodes reading batches of training samples at random, where data is not reused soon enough for a read cache or prefetching to help. That early version turned off the file cache, read with Direct I/O through Linux asynchronous I/O interfaces, aligned buffers inside the file system so users did not have to, and asked users to pack samples into FFRecord, a High-Flyer binary format with PyTorch Dataset and DataLoader adapters. The post said Fire-Flyer 2 then used 64 storage servers for 3FS, each with sixteen 15 TB SSDs and two high-speed NICs, and that data reading took about 1.8% of total epoch time in distributed [ResNet](https://aiwiki.ai/wiki/resnet) training even with the cluster fully loaded.[8] A follow-up post on server-side optimization said 3FS reads with Direct I/O and RDMA Read so that sample loading uses very little CPU and memory, and that the client converts synchronous reads above a threshold (1 MB by default) into asynchronous ones.[9]

### Fire-Flyer AI-HPC paper

DeepSeek and High-Flyer engineers described the full Fire-Flyer 2 system in "Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning", posted to arXiv on 26 August 2024 and accepted at SC24, the 2024 International Conference for High Performance Computing, Networking, Storage, and Analysis. Its 52 authors include [Liang Wenfeng](https://aiwiki.ai/wiki/liang_wenfeng) (credited as Wenfeng Liang), and its acknowledgments thank colleagues at both DeepSeek-AI and High-Flyer Quant for four years of design, construction and operation.[3] The paper says Fire-Flyer 2 has 10,000 PCIe A100 GPUs and reached performance close to the DGX-A100 at half the cost and with 40% less energy. It presents 3FS as part of a software stack that also includes the HFReduce allreduce library, the HaiScale parallelism framework and the HAI Platform scheduler.[3] The paper compares 3FS to WekaFS, DAOS and BeeGFS, and says its design differs by focusing on fully using the high IOPS and throughput of NVMe SSDs and the RDMA network.[3]

### Open-source release

DeepSeek announced a week of daily open-source releases starting 24 February 2025: [FlashMLA](https://aiwiki.ai/wiki/flashmla) on day one, [DeepEP](https://aiwiki.ai/wiki/deepep) on day two, [DeepGEMM](https://aiwiki.ai/wiki/deepgemm) on day three, [DualPipe](https://aiwiki.ai/wiki/dualpipe) and EPLB on day four, and 3FS on day five, under the heading "3FS, Thruster for All DeepSeek Data Access".[4] The day-five post on X went out at 01:06 UTC on 28 February 2025.[5] The GitHub repository's initial commit is dated 27 February 2025, and outside contributors began sending fixes the following day.[20]

| Date | Event | Source |
|---|---|---|
| 2019 | High-Flyer registers an AI research company; Fire-Flyer 1 built with 500 GPUs | [7] |
| 2020 | Fire-Flyer 1 in service with 1,100 accelerator cards | [7] |
| 2021 | High-Flyer invests one billion yuan in Fire-Flyer 2, whose first phase includes 3FS | [7] |
| 2 Aug 2024 | DeepSeek API launches context caching on disk | [10] |
| 26 Aug 2024 | Fire-Flyer AI-HPC paper (SC24) posted to arXiv, describing 3FS and 3FS-KV | [3] |
| 28 Feb 2025 | 3FS and smallpond open-sourced on day five of Open Source Week | [4][5] |
| 31 Jul 2025 | SGLang merges a 3FS ("hf3fs") storage backend for its hierarchical KV cache | [16] |
| 26 Apr 2026 | DeepSeek-V4 technical report says DSec runs on top of 3FS | [12] |
| 19 Sep 2026 | DSec paper details how sandbox images load on demand from 3FS | [13] |

## Architecture

3FS has four components: a cluster manager, a metadata service, a storage service and a client, all connected over an RDMA network.[2] Applications reach storage "in a locality-oblivious manner": data is striped across the SSDs of all storage nodes rather than placed near the compute node that uses it, a design the README calls a disaggregated architecture.[1]

| Component | Role | Notes |
|---|---|---|
| Cluster manager (mgmtd) | Tracks membership, distributes cluster configuration and chain tables | Several run with one elected primary; DeepSeek stores their configuration in the same key-value store as file metadata to reduce dependencies |
| Metadata service (meta) | Implements file system semantics (open, create, rename, list) | Stateless; all state lives in FoundationDB, so clients can connect to any instance |
| Storage service | Manages a few local SSDs and exposes a chunk store | Implements CRAQ replication |
| Client | FUSE client for most applications, native USRBIO API for performance-critical ones | The native client runs inside the FUSE daemon |

Sources: design notes and setup guide.[2][19] The reference deployment also uses ClickHouse to store metrics collected by a monitor service.[19]

### Metadata on FoundationDB

All file system metadata is stored as key-value pairs in FoundationDB, which provides transactions with Serializable Snapshot Isolation. Inode keys combine an "INOD" prefix with a 64-bit inode ID encoded little-endian so that inodes spread across FoundationDB nodes. Directory entry keys combine a "DENT" prefix, the parent inode ID and the entry name, which makes every directory a contiguous key range that can be listed with a range query. Queries such as stat, lookup and listdir run as read-only transactions, and updates such as create, link, unlink and rename run as read-write transactions that the metadata service retries automatically on conflict.[2]

DeepSeek argues for a file interface over an object store because its applications need atomic directory moves, recursive deletes, and symbolic and hard links. A common internal pattern writes files into a temporary directory and then moves the whole directory into place, and hard links are used to snapshot datasets that grow by appending files.[2] To keep FoundationDB load down, 3FS does not track file descriptors opened read-only, since training jobs open very large numbers of files at startup. It keeps a session only for files open for writing, and file lengths are only eventually consistent while several clients write: clients report their maximum write position every 5 seconds by default, and the exact length is computed on close or fsync.[2]

### Chunks, chains and CRAQ

A file is split into equal-sized chunks striped across several replication chains, and users can set the chain table, chunk size and stripe size per directory. When a file is created, the metadata service picks consecutive chains round-robin from the chain table and shuffles them with a random seed. Clients then compute chunk IDs and chain locations themselves, which keeps the metadata service off the critical data path. DeepSeek's production stripe size is 200.[2]

Each chunk is replicated along a chain of storage targets using CRAQ, a write-all-read-any protocol. Writes enter at the head and propagate to the tail; reads can go to any target, which lets 3FS use the read bandwidth of every replica.[2] A storage service pulls write data from the client or predecessor with RDMA Read, serializes writes to a chunk at the head, stores the update as a pending version, and commits it once the tail acknowledges. 3FS departs from textbook CRAQ on reads: instead of asking the tail for the committed version, a target holding both committed and pending versions returns a special status, and the client can wait and retry or ask for a "relaxed read" of the pending version.[2]

Every SSD hosts several storage targets that belong to different chains. The design notes show how chain tables can be arranged so that when one SSD fails its read traffic is spread across many other SSDs rather than the two neighbours in its chain, and say the optimal layout is found by treating it as a balanced incomplete block design solved with an integer programming solver.[2] Failed targets are moved to the end of their chain, and on return they are brought up to date through full-chunk-replace writes from their predecessor while normal traffic continues. Failure detection uses heartbeats as leases: a service that cannot reach the cluster manager for half the timeout stops serving and exits.[2]

On each SSD, a chunk engine stores chunk data in a fixed set of data files and chunk metadata in RocksDB. Updates are copy-on-write, with physical blocks in 11 power-of-two sizes from 64 KiB to 64 MiB, and appends are written in place.[2] The chunk engine is implemented in Rust inside a code base that is otherwise mostly C++.[20]

### FUSE and the USRBIO native client

Most applications use the FUSE client. The design notes list two FUSE limits: memory copies between kernel and user space, and a spin-locked request queue that in DeepSeek's benchmark capped FUSE at about 400,000 4 KiB reads per second. FUSE on Linux 5.x also does not support concurrent writes to the same file, so applications write to many files in parallel instead.[2] Rather than write a kernel module, which DeepSeek considered too risky to debug and upgrade in production, the team built a native client inside the FUSE daemon with an asynchronous zero-copy API modelled on Linux io_uring. Applications open a file normally, register the descriptor, and submit requests through a shared ring buffer (Ior) against a shared, RDMA-registered memory region (Iov); metadata operations still go through FUSE.[2] This interface is documented as the USRBIO API, and DeepSeek ships a fio engine for it for benchmarking.[1]

### Network co-design in Fire-Flyer 2

In Fire-Flyer 2, storage and compute traffic share the same [InfiniBand](https://aiwiki.ai/wiki/infiniband) fabric: two zones, each a two-layer fat-tree, with every storage server connected to both zones through its two NICs so that all compute nodes can reach the same storage services. The paper describes separating HFReduce, NCCL, 3FS and other traffic onto different InfiniBand service levels and virtual lanes, and using static rather than adaptive routing because adaptive routing spread congestion under 3FS's incast-heavy traffic.[3] 3FS itself adds a request-to-send mechanism: after a storage service reads data from SSD it asks the client for permission to send, the client limits the number of concurrent senders, and the data then arrives by RDMA WRITE followed by an RDMA SEND. The paper says this raises end-to-end latency but is needed for sustained high throughput.[3]

### Formal specifications

The repository includes specifications written in the P modelling language. The DataStorage specification models the CRAQ implementation in 3FS and is tested with one to three clients, with and without failures; a second specification checks the RDMA socket implementation.[18]

## Performance

The README reports three benchmarks, each with its own hardware setup.[1]

| Test | Setup | Result |
|---|---|---|
| Peak read throughput | 180 storage nodes, each with 2x200 Gbps InfiniBand NICs and sixteen 14 TiB NVMe SSDs; about 500+ client nodes, each with 1x200 Gbps InfiniBand NIC; background traffic from training jobs | About 6.6 TiB/s aggregate read throughput |
| GraySort with smallpond | 25 storage nodes (2 NUMA domains per node, 1 storage service per NUMA domain, 2x400 Gbps NICs per node) and 50 compute nodes (192 physical cores, 2.2 TiB RAM, 1x200 Gbps NIC each) | 110.5 TiB sorted across 8,192 partitions in 30 minutes 14 seconds, an average of 3.66 TiB/min |
| KVCache reads | KVCache client nodes with 1x400 Gbps NIC each | Peak read throughput up to 40 GiB/s |

The GraySort run used a two-phase approach, a shuffle that partitions data by key prefix followed by in-partition sorting, with both phases reading from and writing to 3FS.[1] The Open Source Week announcement gives the KVCache figure as "40+ GiB/s peak throughput per client node".[4]

The Fire-Flyer AI-HPC paper describes the same 180-node storage tier in slightly different terms: each node has one 64-core AMD EPYC 7742 CPU, 512 GB of DDR4-3200 memory, two Mellanox ConnectX-6 200 Gbps InfiniBand NICs and sixteen 15.36 TB PCIe 4.0 NVMe SSDs. Its 360 NICs give about 9 TB/s of outbound bandwidth, of which the paper says 8 TB/s of total read throughput was achieved, and the 2,880 SSDs provide over 20 PiB with mirrored redundancy.[3] The two figures are not directly comparable: the paper reports 8 TB/s without describing the test or saying whether it means decimal or binary units, while the README's figure of about 6.6 TiB/s was measured with training traffic running in the background.[1][3]

## Uses at DeepSeek

The README lists four workload classes: organizing the outputs of data-analytics pipelines into directory trees, dataloaders that read random training samples across compute nodes without prefetching or shuffling, high-throughput parallel checkpointing, and KV cache storage for inference as a cheaper, larger alternative to DRAM.[1]

- **Checkpointing.** The Fire-Flyer paper describes a checkpoint manager that splits parameters and optimizer state into chunks and writes them with the 3FS batch write API at over 10 GiB/s per node. Checkpoints are saved asynchronously, typically every 5 minutes, and a load finishes in a few seconds using the batch read API, so a hardware failure costs only the last 5 minutes of training.[3]
- **3FS-KV and context caching.** The same paper describes 3FS-KV, a shared-storage data processing system on top of 3FS that supports key-value, message-queue and object-storage models, and says it supports DeepSeek's "KV Context Caching on Disk".[3] DeepSeek launched that feature on its API on 2 August 2024, describing a cache on "a distributed disk array" that serves repeated prompt prefixes without recomputation.[10] In DeepSeek's day-six Open Source Week post, 342 billion of 608 billion input tokens (56.3%) served by V3 and R1 over one 24-hour period in February 2025 hit the on-disk KV cache.[14]
- **DeepSeek-V3 training cluster.** In DeepSeek's ISCA 2025 paper on the hardware behind [DeepSeek-V3](https://aiwiki.ai/wiki/deepseek_v3), each node in the H800 training network has a 400 Gbps Ethernet RoCE NIC on a separate storage network plane used to reach 3FS.[11]
- **DSec sandbox images.** The [DeepSeek V4](https://aiwiki.ai/wiki/deepseek_v4) technical report says the three Rust components of DSec, DeepSeek's sandbox platform for agentic training, "scale horizontally atop the 3FS distributed filesystem".[12] The DSec paper explains that DeepSeek hosts container and microVM images on 3FS instead of running a container registry with peer-to-peer delivery. Because 3FS "sustains high throughput for large sequential reads and writes but performs poorly on small random I/O", DSec keeps sandbox writes on local disk, fetches read-only image data from 3FS on demand in bulk, and keeps filesystem metadata local where it can. The paper describes 3FS storage servers in that deployment with twenty 15 TB SSDs and two 400 Gbps RDMA NICs each, mounted by CPU nodes through the FUSE client, and says tens of such servers support on-demand image loading for hundreds of thousands of CPU cores.[13]

## smallpond

smallpond is a Python data processing framework "built on DuckDB and 3FS", released alongside 3FS under the MIT License.[6] It runs SQL and Python transformations on partitioned datasets, typically Parquet files, uses DuckDB for execution and Ray Core as its task scheduler, and needs no long-running services. Its README says it scales to petabyte-scale datasets.[6][21] Users partition data by hand, for example by file, by row or by a hash of a column.[21] The GraySort result in the 3FS README (110.5 TiB in 30 minutes 14 seconds) was produced with smallpond.[1][6] Version 0.15.0 was uploaded to PyPI on 28 February 2025.[22]

## License and development

3FS is released under the MIT License, copyright 2025 DeepSeek.[1] Building it requires libfuse 3.16.1 or newer, FoundationDB 7.1 or newer and a Rust toolchain, and the README gives dependency lists for Ubuntu 20.04 and 22.04, openEuler and OpenCloudOS 9 / TencentOS 4, with ready-made build images for the last two.[1] Changes merged in the weeks after release added AArch64 builds, support for bonded NICs and Rust bindings.[20] Since late 2025 builds have had to pin a "shuffle method" (`g++10` or `g++11`), because binaries built with different compiler versions used different `std::shuffle` implementations and could be incompatible within one cluster.[1] Commits continued into 2026, with the most recent on 7 May 2026 fixing chain table version ordering.[20] On 23 September 2026 the repository had about 10,200 GitHub stars and 1,100 forks.[1]

## Adoption outside DeepSeek

The inference engine [SGLang](https://aiwiki.ai/wiki/sglang) supports 3FS as a storage backend, named `hf3fs`, for HiCache, its hierarchical KV cache. The backend was merged on 31 July 2025, and its documentation describes 3FS as an "L3" KV cache shared across instances.[16] The SGLang team credits engineers from Alibaba Cloud's TairKVCache team with the integration, and its HiCache announcement quotes the cloud provider Novita AI: using HiCache with 3FS on a Qwen3-Coder-480B coding-agent workload cut average time to first token by 56%, doubled throughput and raised the cache hit rate from 40% to 80%.[15] Alibaba Cloud also publishes kvc-3fs-operator, a Kubernetes operator that deploys 3FS components and injects FUSE sidecars into application pods.[17] See [KV cache offloading](https://aiwiki.ai/wiki/kv_cache_offloading) and [RadixAttention](https://aiwiki.ai/wiki/radix_attention) for how such storage tiers are used in serving.

## Limitations

DeepSeek's own documents describe several constraints. FUSE access tops out at roughly 400,000 small reads per second and does not allow concurrent writes to one file on Linux 5.x, so high-performance small reads need the USRBIO native API.[2] File lengths are only eventually consistent during concurrent writes, and 3FS does not track descriptors opened read-only, whereas most local file systems defer deleting an open file until every descriptor is closed.[2] The DSec paper says 3FS performs poorly on small random I/O, which is why DSec keeps writes and most metadata on local disk.[13] Deployment also needs RDMA-capable NICs and a separately operated FoundationDB cluster.[19]

## References

1. [deepseek-ai/3FS: Fire-Flyer File System (README and repository), GitHub](https://github.com/deepseek-ai/3FS)
2. [3FS Design Notes, deepseek-ai/3FS, GitHub](https://github.com/deepseek-ai/3FS/blob/main/docs/design_notes.md)
3. [An, W. et al. "Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning." arXiv:2408.14158 (SC24), 2024](https://arxiv.org/abs/2408.14158)
4. [DeepSeek Open Infra Index (Open Source Week, February 2025), GitHub](https://github.com/deepseek-ai/open-infra-index)
5. [DeepSeek (@deepseek_ai). "Day 5 of #OpenSourceWeek: 3FS, Thruster for All DeepSeek Data Access." X, 28 February 2025](https://x.com/deepseek_ai/status/1895279409185390655)
6. [deepseek-ai/smallpond, GitHub](https://github.com/deepseek-ai/smallpond)
7. [幻方历程 (High-Flyer history), High-Flyer](https://www.high-flyer.cn/history/)
8. [幻方力量 | 高速文件系统 3FS (High-Flyer technical blog)](https://www.high-flyer.cn/blog/3fs/)
9. [3FS优化 01 | 服务端优化 (High-Flyer technical blog)](https://www.high-flyer.cn/blog/3fs-1/)
10. ["DeepSeek API introduces Context Caching on Disk, cutting prices by an order of magnitude." DeepSeek API Docs, 2 August 2024](https://api-docs.deepseek.com/news/news0802)
11. [Zhao, C., Deng, C., Ruan, C. et al. "Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures." arXiv:2505.09343 (ISCA 2025)](https://arxiv.org/abs/2505.09343)
12. [DeepSeek-AI. "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence." arXiv:2606.19348, 2026](https://arxiv.org/abs/2606.19348)
13. [Huang, J. et al. "DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scale." arXiv:2609.22978, 2026](https://arxiv.org/abs/2609.22978)
14. [Day 6: DeepSeek-V3/R1 Inference System Overview, deepseek-ai/open-infra-index, GitHub](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md)
15. ["SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends." LMSYS Blog, 10 September 2025](https://www.lmsys.org/blog/2025-09-10-sglang-hicache/)
16. [SGLang hf3fs storage backend and documentation, sgl-project/sglang, GitHub](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/hf3fs)
17. [aliyun/kvc-3fs-operator, GitHub](https://github.com/aliyun/kvc-3fs-operator)
18. [3FS P specifications, deepseek-ai/3FS, GitHub](https://github.com/deepseek-ai/3FS/blob/main/specs/README.md)
19. [3FS Setup Guide, deepseek-ai/3FS, GitHub](https://github.com/deepseek-ai/3FS/blob/main/deploy/README.md)
20. [Commit history, deepseek-ai/3FS, GitHub](https://github.com/deepseek-ai/3FS/commits/main)
21. [smallpond Getting Started, deepseek-ai/smallpond, GitHub](https://github.com/deepseek-ai/smallpond/blob/main/docs/source/getstarted.rst)
22. [smallpond, Python Package Index](https://pypi.org/project/smallpond/)