JuiceFS Blog

JuiceFS Performance Optimization for AI Scenarios

Wed, 15 Apr 2026 07:22:00 +0000

The scale of computing power for large language model (LLM) training continues to expand. While GPU performance keeps improving, data access bottlenecks are becoming increasingly prominent in overall system performance. Local storage offers excellent performance but has limited scalability. Object storage excels in cost and scalability but suffers from insufficient throughput in massive small‑file and high‑concurrency scenarios. Teams often struggle to choose between them.

Therefore, distributed file systems have become a key solution to balance high performance and scalability. JuiceFS has been widely deployed in AI scenarios across multiple industries. Its distributed architecture delivers high performance, strong scalability, and low cost simultaneously for large‑scale data access.

In this article, we’ll introduce JuiceFS’ architecture from a performance perspective and analyze core performance bottlenecks and optimization methods under different access patterns. We’ll also offer links of key points for references, helping you understand JuiceFS’ performance mechanisms and master common tuning strategies.

Performance foundations from the JuiceFS architecture

JuiceFS comes in Community Edition and Enterprise Edition. Both share the same architecture: metadata and data are separated. The client adopts a rich‑client design, handling core logic including some metadata operations, and provides both metadata and data caching. These modules work together for efficient data location and access. The underlying data is stored in object storage, with local caches further improving access performance. For external interfaces, JuiceFS supports multiple access methods – FUSE is the most common, and it also provides various SDKs and an S3 gateway.

JuiceFS Community Edition is designed as a general‑purpose file system. Users can choose different metadata engines based on their needs. For small‑scale deployments, Redis delivers lightweight, low‑latency metadata management. For large‑scale file scenarios, TiKV provides good horizontal scalability.

JuiceFS Enterprise Edition targets complex, high‑performance scenarios. It differs from Community Edition in two ways:

It uses a self‑developed multi‑zone metadata engine built on Raft that runs as an in‑memory cluster, offering low latency and strong horizontal scalability. It supports up to 500 billion files. Operations that require multiple key-value requests in the Community Edition often need only one or two in the Enterprise Edition, and complex logic can be processed inside the metadata cluster.
The Enterprise Edition supports distributed cache sharing: clients in the same group can access each other’s local caches via consistent hashing. This improves cache hit rates and access efficiency. In multi‑node, high‑concurrency scenarios, the cache space scales horizontally, and most required data can be warmed up before job execution. This accelerates AI training and inference while boosting performance and stability. See JuiceFS Enterprise 5.3: 500B+ Files per File System & RDMA Support.

image: JuiceFS 社区版和企业版架构
caption: JuiceFS Community Edition and Enterprise Edition architectures

Data chunking

JuiceFS splits data into chunks and stores them in object storage. This design is key to its performance, affecting data read efficiency, cache hit rate, and throughput under high concurrency.

JuiceFS breaks a file into multiple chunks. Inside each chunk, the system maintains a management structure called a slice to track writes and updates. When data is written, new data does not overwrite existing slices; instead, a new slice is appended on top of the chunk.

image: chunk
caption: How JuiceFS stores data

Ideally, each chunk ends up containing only one slice. Each slice consists of several 4 MB blocks, which are the smallest unit stored in object storage. By default, the caching system also manages data at the block level.

image: block
caption: How JuiceFS stores data

As shown in the diagram on the upper right, file updates use an append‑only write pattern: existing slices are shown in red, and new data is appended as a new slice. During reads, the system combines the slices to form the current view. When fragmentation becomes excessive, a compaction process merges slices to optimize access performance. For more details on data chunking, refer to Code-Level Analysis: Design Principles of JuiceFS Metadata and Data Storage.

Caching

Compared to direct object storage access, JuiceFS performance improvements largely benefit from its caching mechanism. The JuiceFS client comes with a high‑performance local cache module. Key configuration options include:

cache-dir: specifies the cache directory.
cache-size: sets the maximum cache space.
Prefetch: a parameter in the cache module that controls prefetching. When a request hits a block, a background thread fetches the entire block.
Write‑back related settings: improves write IOPS by writing data blocks that need to be uploaded to object storage into the local cache first, then asynchronously uploading them to object storage.

JuiceFS Enterprise Edition also provides advanced configurations. For example, a cache group can be used to designate a set of clients whose local caches form a distributed cache group, enabling cache sharing. In addition, the no sharing option (related to cache groups) allows a client to read data only from a specified cache group without serving its own cache to others. This creates a two‑level cache:

The first level is the local cache.
The second level is the cache on other nodes in the group.

Another performance‑boosting mechanism is the memory buffer (read buffer), which provides:

I/O request merging: multiple consecutive I/O requests can be merged in memory. For example, three I/O requests issued by the system may be reduced to just one after being processed by the memory buffer.
Adaptive read‑ahead: in large‑file sequential read scenarios, adaptive read‑ahead increases request concurrency by prefetching data. This fully utilizes cache and object storage resources and improves overall I/O performance.

The Enterprise Edition also offers advanced read‑ahead settings:

max read ahead: sets the maximum read‑ahead range.
initial read ahead: sets the initial read‑ahead window size (default unit is 4 MB blocks).
read ahead ratio: a configuration added last year that controls the read‑ahead ratio for large‑file random reads, reducing bandwidth waste caused by read amplification. Overly aggressive read‑ahead can negatively impact random read performance; read ahead ratio helps mitigate this. In AI scenarios, when large‑file sequential or random reads cause bandwidth or IOPS bottlenecks, adjusting these parameters can optimize overall performance.

JuiceFS benchmark I/O tests and bottleneck analysis

Before diving into performance tuning for common AI scenarios, let’s first examine JuiceFS’ I/O behavior under ideal conditions through sequential and random read benchmarks. This helps us understand throughput and latency under different access patterns, providing a reference for the read/write patterns of subsequent AI/ML workloads.

Sequential read performance

In JuiceFS, sequential read performance is typically bandwidth‑bound. In cold read scenarios, performance is mainly limited by object storage bandwidth; in distributed cache scenarios, network bandwidth can become the bottleneck. For example, a node with a 40 Gbps NIC may achieve less than 5 Gbps usable bandwidth. In addition, the user‑kernel transition overhead in the FUSE layer limits single‑thread throughput. Tests showed single‑thread sequential read bandwidth around 3.5 Gbps. To break this limit, multi‑threaded or higher‑concurrency strategies are needed to fully utilize storage and network resources.

The table below shows test results of JuiceFS sequential read performance:

Threads	Bandwidth (GB/s)	Bandwidth per thread (GB/s)
1	3.5	3.5
2	6.3	3.15
3	9.5	3.16
4	9.7	2.43
6	14.0	2.33
8	17.0	2.13
10	18.6	1.9
15	21	1.4

In the performance test, single‑thread sequential read bandwidth was about 3.5 Gbps. As the number of threads increased, total throughput gradually approached the network bandwidth limit. To help users evaluate the performance ceiling of their own environment, JuiceFS provides the bj bench subcommand for testing object storage bandwidth.

In real workloads, caching is more common than direct object storage access. In such cases, increasing the buffer size raises the number of background prefetch requests, thereby improving concurrency and overall throughput. For example, after increasing the buffer size to 400 MB (corresponding to 100 background prefetch requests of 4 MB each), concurrency improved significantly and overall throughput increased.

Random read performance

Low‑concurrency random reads

In low‑concurrency, non‑asynchronous access scenarios, each request must wait for the previous one to complete before being issued. As a result, latency has a significant impact on overall performance. I/O latency can come from many sources, including metadata query latency, object storage access latency, and local or distributed cache read latency. When analyzing random read performance, we must closely examine these latency factors.

In a 4 KB cold random read scenario, if the IOPS is only 8 and object storage latency is about 125 ms, the concurrency level is roughly 1 (8 IOPS × 125 ms ≈ 1,000 ms).

This indicates a near‑single‑concurrent, serial‑blocked state. In such cases, the optimization focus should be on shortening the access path and reducing per‑request latency rather than increasing concurrency – for example, by warming up data into the local cache. After data warm-up, the random read path switches from object storage to local cache, and IOPS can increase to about 12,000, approaching the I/O level of a local disk.

image: juicefs stats 命令查看性能
caption: Using the juicefs stats command to view performance

image: 预热后性能
caption: Performance after data warm-up

High‑concurrency random reads

High‑concurrency random reads typically occur in scenarios with high thread counts or asynchronous I/O. The main performance bottleneck is often IOPS limits – including metadata IOPS, object storage IOPS, and cache IOPS. JuiceFS allows you to observe these metrics and pinpoint the bottleneck. Client machine resources (CPU, memory) can also affect performance, but such bottlenecks are easy to monitor.

In a cold read scenario using Libaio for random reads, the object‑side IOPS ceiling is around 7,000/s. When caching is enabled and data is warmed up, the access path shifts from object storage to the cache layer, and IOPS can further increase to over 20,000. This shows that the bottleneck for high‑concurrency random reads shifts as the access path changes.

image: 预热前
caption: Before data warm-up

image: 预热后
caption: After data warm-up

For a deeper dive into JuiceFS’ complete data access path, refer to Optimizing JuiceFS Read Performance: Readahead, Prefetch, and Cache.

I/O characteristics and performance tuning for common AI scenarios

Large‑file sequential reads

A typical large‑file sequential read scenario is model loading, such as loading PyTorch .pt files saved via pickle serialization. In this process, performance is limited by two factors:

Pickle deserialization efficiency determines data processing speed.
Data reading is usually single‑threaded and limited by FUSE bandwidth and CPU performance.

To increase throughput, you can raise concurrency through multi‑threaded or sharded loading, fully utilizing I/O capacity. For large‑file sequential reads, the best performance is achieved when the entire dataset can be cached locally. If only on‑demand reading is required, the implementation is simple.
For more details on optimizing large‑file sequential reads, see How JuiceFS Transformed Idle Resources into a 70 GB/s Cache Pool.

Massive small files

In computer vision and multimodal tasks, training datasets often consist of many individual files, for example, single images, video frames, or text annotations. Such massive small‑file scenarios place heavy pressure on metadata services.

In massive small-file scenarios, metadata performance is critical. On one hand, each file carries only a small amount of data; on the other hand, directory metadata access efficiency is low when a directory holds a huge number of small files.
For read‑only workloads, enabling client metadata caching and extending the cache lifetime can improve performance.

Moreover, the data read layer experiences higher IOPS pressure because small files cannot take advantage of read‑ahead. This makes requests more fragmented. Common optimizations include increasing local cache capacity; for the Enterprise Edition, you can also scale out the distributed cache cluster horizontally. Because small files derive little benefit from read‑ahead, their latency tends to be higher.

For performance tuning in this scenario, see How D-Robotics Manages Massive Small Files in a Multi-Cloud Environment with JuiceFS.

Large‑file random reads

This scenario is common in AI training, for example, when randomly accessing datasets in TFRecord, HDF5, or LMDB format by sample. Take model loading: if the dataset is accessed randomly and each read size equals the sample size (for example, 1 MB to 4 MB images or short videos), read‑ahead can waste bandwidth. Such scenarios can often break through IOPS bottlenecks by increasing concurrency.

Recommended measures include:

Increase the number of data‑loading reader threads.
Use asynchronous I/O to raise concurrency and saturate IOPS.
Improve the caching system, for example, pre‑map data into cache to boost underlying IOPS.
Adjust the read ahead ratio parameter (for example, set it to 0.5) to reduce bandwidth waste from read‑ahead. For instance, a 4 MB sequential read would previously prefetch 4 MB; after adjustment, only 2 MB is prefetched.

In this article, we’ve analyzed JuiceFS’ architecture from a performance perspective, covered benchmark I/O tests, and discussed tuning methods for typical AI scenarios. This provides an introductory reference for system performance. JuiceFS has been deployed in many production environments, and its distributed architecture offers a feasible balance between performance and cost.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

Optimizing JuiceFS on the Arm Architecture: MLPerf-Based Performance Tuning

Thu, 02 Apr 2026 12:45:00 +0000

Recently, building high-performance storage infrastructure on Arm platforms has become a technical focal point. Linaro is an international technology organization focused on the Arm ecosystem and open-source software. We collaborate with upstream and downstream industry players to address common issues and assist enterprise customers in productizing their solutions on an open-source foundation. Our team conducted systematic stress testing on JuiceFS Community Edition (using Redis for metadata storage) during MLPerf Storage benchmarks, covering a variety of typical machine learning training workloads.

Our test results show that system performance is largely influenced by memory bandwidth and metadata access efficiency. JuiceFS’ throughput directly determines GPU utilization and training efficiency. Through testing workloads such as 3D U-Net, ResNet-50, and CosmoFlow, the analysis revealed: in single-node scenarios, GPU utilization is primarily limited by memory copy latency; in two-node or multi-node scenarios, metadata access and inter-node synchronization become the main bottlenecks. In the article, we also provide tuning strategies and practical results to address these bottlenecks.

In summary, large-scale AI training performance tuning is a systematic engineering effort that requires coordinated optimization across storage systems, memory bandwidth, CPU scheduling, caching strategies, and more to achieve efficient deep learning data supply on Arm platforms.

Arm64 vs. x86_64 architecture differences and concurrency characteristics

Compared to x86, Arm’s application scope continues to expand, extending from mobile devices to IoT, wearables, PCs, automotive, and servers. Its high performance per watt is a key reason for its widespread adoption.
From an architectural design perspective, Arm is a reduced instruction set computer (RISC), while x86 is a complex instruction set computer (CISC). This design difference also affects how processors execute instructions. Arm64 instructions have a fixed length of 4 bytes, whereas x86 instructions have variable lengths ranging from 1 to 15 bytes. Consequently, x86 often requires more complex decoders. In contrast, Arm’s instructions are simpler and rely more heavily on effective instruction organization during compilation and code generation, thus requiring longer compilation times.

From an engineer’s perspective, there are other architectural differences that directly impact program behavior. Code that seems intuitive on x86 may not behave the same way on Arm. Several of the common pitfalls discussed later are fundamentally related to these underlying differences.

One typical issue is the alignment requirement for atomic operations. Whether using Load-Link/Store-Conditional (LL/SC) or Large System Extensions (LSE), read-modify-write operations like atomic increments typically require aligned memory addresses. Newer LSE2 relaxes this restriction, supporting unaligned accesses within a 16-byte window. Data alignment is not mandatory for x86, but maintaining good alignment helps improve performance. See Arm Architecture Reference Manual for A-profile architecture.

Another key feature to note is that Arm employs a weakly ordered / relaxed memory model. The difference lies in the strength of constraints on memory access ordering. In multi-threaded scenarios, the same read/write operations are more likely to appear in program order on x86, whereas Arm permits more reordering. Thus, the order observed by other threads may differ from the source code order. When debugging issues on Arm, memory ordering effects must be carefully considered. For more details, see the Arm white paper: Synchronization Overview and Case Study on Arm Architecture.

Overview of JuiceFS and MLPerf

JuiceFS is an open-source, high-performance distributed file system built on object storage. It leverages the cost advantages of object storage while delivering a user experience close to traditional file systems. It supports POSIX, HDFS SDK, Python SDK, and S3-compatible interfaces, adapting to various applications and data processing frameworks. It also supports cloud-native extensions, data security, and compression, making it widely applicable to AI training, inference, big data processing, and more.

image: 1JuiceFS 架构图.drawio
caption: JuiceFS Community Edition architecture

To evaluate JuiceFS’ data supply capability under high-load scenarios like AI training, we can use the MLPerf Storage benchmark. Developed by MLCommons, this benchmark focuses on measuring a storage system’s ability to consistently and efficiently supply data to compute nodes.

Version 2.0 divides tests into training workloads and checkpoint workloads. The training workloads include 3D U-Net, ResNet-50, and CosmoFlow. They differ significantly in sample size and access patterns. Minimum GPU utilization requirements are set: 90% for 3D U-Net and ResNet-50, and 70% for CosmoFlow.

The table below shows MLPerf Storage 2.0 training workloads:

Task	Reference network	Data loader	Sample size	Batch size	Accelerator utilization	Time per batch run (s)	Evaluate storage capability
Image segmentation (medical)	3D U-Net	PyTorch	146 MiB	7 x 146 = 1,022 MiB	90%	0.323 / 0.9 = 0.359 Data load time: 0.359-0.323 = 0.036	Bandwidth, concurrent large block sequential reads
Image classification	ResNet50	Tensorflow	150 KiB	400 x 150 = 58.5 MiB	90%	0.224 / 0.9 = 0.249 Data load time: 0.249 - 0.224 = 0.025	Bandwidth, IOPS, high concurrency medium block sequential reads
Scientific (cosmology)	Parameter prediction	Tensorflow	2.7 MiB	1 x 2.7 = 2.7 MiB	70%	0.0035 / 0.7 = 0.005 Data load time: 0.005 - 0.0035 = 0.0015	Bandwidth, IOPS, metadata latency, high concurrency sequential reads of many small files
LLM checkpointing (new)	Llama3	PyTorch	105GiB to 18TiB	—	—	—	Bandwidth, concurrent sequential writes of extremely large files

In the test flow, data is first read from the storage system into host memory before entering the compute phase. Training time is simulated to replicate the data flow of real training scenarios, eliminating the need for actual GPU deployment, lowering experimental barriers, and improving operational convenience.

image: MLPerf Storage 数据流
caption: MLPerf Storage data flow

MLPerf Storage v2.0 test principles and tuning

Before detailing specific model test results, it’s essential to understand the data access principles of distributed training. This helps readers grasp the causes of GPU utilization, storage throughput, and performance bottlenecks, enabling better comprehension of subsequent test results and tuning strategies.

Distributed machine learning typically uses data parallelism, where multiple parallel processes share the same dataset, and each process handles reading and processing its corresponding training batches.

image: 分布式训练数据访问原理示意图
caption: Distributed training data access principle

MLPerf Storage training tests follow this approach: each training process reads data from the storage system in batches and simulates computation to evaluate the storage system’s ability to sustain data supply.

image: MLPerf Storage 训练数据流示意图
caption: MLPerf Storage training data flow

To understand the source of performance during testing, it’s also necessary to understand the data processing path within the JuiceFS client.

image: JuiceFS 客户端线程与数据流示意图
caption: JuiceFS client threads and data flow

As illustrated, when testing with JuiceFS, the execution flow can be roughly divided into three parts:

Left side: Application-side I/O threads, such as fio or MLPerf Storage’s DataLoader threads, which initiate read/write requests and wait for completion.
Middle: The main goroutine in the FUSE daemon, which handles FUSE requests from kernel space, places file data into memory buffers and caches, and triggers backend metadata and object storage access.
Right side: Asynchronous goroutines for the Meta client and ObjectStore client, which interact with the backend MetaDB and ObjectStore clusters for data and metadata operations.

From a performance analysis perspective, we need to note two types of issues:

Data copying, corresponding to steps like 2.1, 3, 4, 5, and 6 in the diagram. These steps introduce additional memory copy overhead and are often key areas for analyzing latency and CPU usage.
Synchronization and asynchronous boundaries. As shown, steps 1, 2, 3, 4, 5, and 6 are part of the synchronous path, where the request must wait for the current stage to complete before proceeding. Step 7 is part of the asynchronous path, handled by background goroutines interacting with backend storage.

Test 1: 3D U-Net

In this test, the sample size was 146 MiB per image file, and we focused on large-block read performance. The test results showed:

In a single-node environment, the system could stably run up to 5 GPUs, with GPU utilization at about 50%.
In a two-node scenario, it could support 10 GPUs, also with GPU utilization around 50%.

To improve data read efficiency, we optimized the training parameters: we increased the number of reader threads from 4 to 16 to accelerate data generation, and switched to direct I/O to reduce buffer and memory copy overhead.
Operational metrics showed that when mounting 6 GPUs on a single node, GPU utilization dropped to 83%, corresponding to a bandwidth of about 15.1 GB/s. This fell short of the expected high utilization target. Further testing with fio on the storage side revealed similar bandwidth of about 15.1 GB/s. This indicated that the bottleneck had shifted to the JuiceFS client bandwidth rather than the GPU compute side.

Optimization analysis 1: CPU pinning

To further investigate the cause of the client bandwidth limitation, we pinned the process to a specific CPU (running on NUMA nodes 2 and 3). Monitoring showed that all 48 CPU cores were nearly fully utilized. Further analysis of top-down, memory, and miss metrics revealed a clear memory-bound condition, with most time spent on memory copying. This indicated that in the CPU-pinned scenario, the performance bottleneck of JuiceFS primarily came from CPU processing capacity and the additional latency caused by cross-NUMA node memory copying.

Optimization analysis 2: no CPU pinning

To understand the bandwidth limitations under more general conditions, we observed the scenario without CPU pinning. The results showed that while the CPU was not fully saturated, the devkit tuner numafast metric indicated that remote memory access accounted for about 80% of total memory accesses. This meant a large number of memory accesses were crossing local NUMA nodes, potentially even across CPU sockets, introducing significant bandwidth loss and access latency.

From the perspective of hardware bandwidth, cross-socket memory access has inherent limitations. For example, on the Arm platform, the theoretical physical bandwidth across sockets was about 60 GB/s. Further measurements showed cross-socket copy bandwidth on Arm1 to be around 48 GB/s, while on two x86 platforms it was about 37 GB/s and 28 GB/s, respectively.

This suggested that in the scenario without CPU pinning, even though the compute cores were not fully exhausted, extensive cross-node, cross-socket remote memory access had become a major source of overhead. Therefore, we inferred that the inability to further increase JuiceFS bandwidth was likely not solely due to CPU compute power, but rather constrained by the bandwidth and latency of cross-socket memory access. In other words, the system bottleneck had shifted from “local CPU being too busy” to “remote memory access being too costly.”

In summary, the reasons for the JuiceFS bandwidth limitation differed between the two scenarios:

With CPU pinning, the bottleneck was primarily CPU resource consumption and the overhead of extensive memory copying.
Without CPU pinning, the bottleneck was largely due to a high proportion of non-local memory accesses, especially the bandwidth and latency penalties from cross-socket accesses.

Test 2: ResNet-50

ResNet-50 uses small samples (about 150 KiB each), with each batch containing 400 samples totaling about 58.5 MiB. This I/O test focused on data loading efficiency and training throughput under high GPU concurrency. The system maintained high utilization at large GPU scales:

Single node: 50 GPUs, 95% GPU utilization, about 9.2 GB/s bandwidth.
Two nodes: 96 GPUs, 90% GPU utilization, about 16.9 GB/s bandwidth.

During testing, we adjusted the reader.read_threads parameter from 8 to 1. For this model (medium-sized images), a single thread sufficed for data supply.

Optimization analysis 1: single-node bottleneck and memory bandwidth impact

With 55 GPUs on a single node, GPU utilization dropped to 86% while bandwidth remained at about 9.2 GB/s. This indicated the bottleneck had shifted to JuiceFS client bandwidth.

Further analysis revealed ResNet-50 tests used buffer I/O mode. Beyond reading data, memory copies during dataset processing consumed part of the memory bandwidth.

System memory copy bandwidth depends on memory channel count, memory frequency, and CPU frequency. Stream tests on nodes with different configurations showed that single-node sequential read bandwidth aligned with measured system memory bandwidth, indicating read throughput largely depends on system memory bandwidth. For training tasks requiring high throughput and GPU utilization, selecting nodes with higher memory bandwidth is recommended to significantly enhance data supply capacity and training efficiency.

	Single-CPU memory copy bandwidth data	JuiceFS single-node deployment read bandwidth
Arm3	Arm3: 171 GB/s	25.3 GiB/s
Arm2	114 GB/s	21.6 GiB/s
Arm1	106 GB/s	18.3 GiB/s
x862	90 GB/s	17.9 GiB/s
x861	82 GB/s	16.6 GiB/s

Optimization analysis 2: two-node scaling bottlenecks and distributed limitations

In multi-node deployments, in addition to single-node performance limits, cross-node memory access, network transfer, and metadata latency become new bottlenecks. Therefore, two-node testing after single-node analysis helped identify these distributed constraints and guide system optimization.

In a two-node scenario, the system theoretically supported up to 100 GPUs, but in actual testing only 96 GPUs could be achieved. Analysis showed that per-operation read latency had increased. Although file data was already cached on local disks, metadata access latency became the primary limiting factor.

To address this issue, we made multiple optimizations:

We grouped CPU cores to ensure training threads and I/O threads ran on the same NUMA node.
Pure data processing and metadata access were assigned to different CPU cores and storage paths.
We adjusted Redis cache and local cache policies to reduce latency under high-concurrency metadata access.

After these tunings, the two-node scenario stably supported 100 GPUs, with GPU utilization reaching the expected level.

Test 3: CosmoFlow

Compared with previous models, this model had a much smaller size per sample. This imposed higher demands on I/O and metadata access. In both single-node and two-node scenarios, the CosmoFlow test showed:

Single node: Stably supported up to 10 GPUs (occasionally up to 12 GPUs), GPU utilization around 75%, bandwidth about 5.6 GB/s.
Key parameter adjustment: reader.read_threads was reduced from 4 to 1, batch size was set to 2 MiB, and a single thread was sufficient to meet data supply requirements.

Optimization analysis 1: single-node bottleneck – memory copy limiting GPU utilization

When we tried to increase the number of GPUs beyond 10, GPU utilization dropped. Log and performance data analysis revealed:

Data read time increased, while metadata access latency did not change significantly.
File data was cached on local disks, disk queues were not full, and latency was low, so the bottleneck was not in the storage device.
Profiling showed that the key bottleneck was memory copy (memcpy) – cumulative delays from multiple copy operations in the data read path increased total read time.

Thus, we inferred that when the system demanded more memory bandwidth, memory copy latency became the main factor limiting read performance and GPU utilization.

Optimization analysis 2: two-node bottleneck – distributed synchronization and metadata latency

In the two-node scenario with 20 GPUs, the first round of testing showed significantly lower GPU utilization. Further analysis found:

One node had started training while the other was still performing dataset preprocessing (reading file lists and sharding).
Because CosmoFlow has a large data volume, reading high-index files took a long time. This caused the two nodes to start training out of sync, leading to lower GPU utilization in the first round.

To resolve this, we added a synchronization mechanism to ensure that all nodes completed dataset preprocessing before starting training. After this adjustment, the two-node test stably supported 20 GPUs, and GPU utilization reached the expected level.

Summary

The key findings and optimization insights from our tests are summarized as follows:

MLPerf Storage evaluates various file system capabilities through different combinations of sample sizes, file sizes, and batch sizes, including large/medium/small sequential read performance, file concurrency, total read bandwidth, metadata access latency, file read latency, and file operation stability. In read-only scenarios, fully utilizing high-speed near-end caches (including data and metadata caches) significantly improved read performance. Note that the smaller the file, the higher the requirements for IOPS and latency.
System memory and bandwidth have a decisive impact on performance. In memory‑copy‑intensive workloads, memory copies consume both bandwidth and CPU cycles, creating the illusion of "CPU busy" while the CPU actually spends most of its time waiting for data. Higher memory bandwidth directly leads to better storage throughput – a key reference for server selection.
The Go runtime has limited NUMA awareness. For large‑core deployments, performance may degrade compared to using fewer cores. Cross‑NUMA (especially cross‑socket) memory accesses should be avoided because cross‑socket bandwidth is typically low (tens of GB/s), increasing latency. In practice, allocate only enough CPU cores, not all, to prevent extra memory access delays.
System‑level optimizations exist. For memory‑copy‑intensive operations, newer Arm systems provide specialized instructions. We collaborated with the Arm community to push configuration improvements, achieving up to tens of percentage points higher bandwidth in some scenarios.
For operations involving heavy kernel‑userspace interaction (for example, file I/O and metadata processing), reducing unnecessary system calls lowers latency. Concentrating file processing within the same production node and avoiding cross‑NUMA/socket access further improves performance and stability.
Cache policy tuning matters. Under high single‑node load, adjusting JuiceFS memory cache policies to reduce invalid memory bandwidth usage effectively increases GPU utilization and storage throughput. Overall, MLPerf Storage Benchmark is a system engineering problem requiring coordinated optimization of file system, memory bandwidth, CPU scheduling, and caching strategies.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

How D-Robotics Manages Massive Small Files in a Multi-Cloud Environment with JuiceFS

Thu, 05 Mar 2026 09:22:00 +0000

D-Robotics, founded in 2024 and spun off from Horizon Robotics' robotics division, specializes in the research and development of foundational computing platforms for consumer-grade robots. In 2025, we released an embodied AI foundation model.

In robot data management, training, and inference, the sheer volume of data is immense. Using object storage presents challenges such as handling small files and managing multi-cloud data. After trying some solutions and replacing private MinIO with SSD storage, we still faced difficulties in addressing these challenges. Ultimately, we selected JuiceFS as our core storage solution.

JuiceFS' inherent adaptability for cross-cloud operations efficiently supports data sharing needs in multi-cloud environments. In training scenarios, JuiceFS' cache mechanism, specifically designed for small file data, effectively replaces traditional caching solutions while achieving a cost-effective balance between cost and efficiency, fully meeting storage performance requirements. Currently, we manage tens of millions of files.

In this article, we’ll share our application characteristics, storage pain points, solution selection, implementation practices, and production tuning experiences. We hope our experience offers useful insights for those facing similar challenges in the industry.

Storage pain points in the robotics industry

The cloud platform serves as our core technical hub, undertaking key application functions such as simulation environment setup, data generation and model training, model lightweighting and deployment, and visual verification. The data types involved in the platform are diverse, mainly including sensor image data, LiDAR point cloud data, model weights and configuration data, motor operational data, and map construction data.

While object storage meets basic storage needs for massive data, its performance limitations become particularly obvious when handling the massive small files frequently encountered in robotics applications. Our storage system faced four challenges:

Metadata performance bottleneck with massive small files: Robot model training involves tens of millions to billions of sensor images, LiDAR data, and model files. Traditional object storage (like standard S3) exhibits significant metadata operation bottlenecks at this scale. The fixed API latency for routine operations like listing files or retrieving attributes is typically 10–30 ms. This directly constrains queries per second (QPS) performance during training and inference and impacts overall R&D efficiency.
Inefficient multi-cloud collaboration and data flow: As robotics companies increasingly adopt multi-cloud architectures for their R&D and production applications, ensuring efficient data synchronization and sharing across different cloud platforms and geographical regions has become a common challenge for the industry. Traditional storage solutions typically suffer from low cross-cloud data transfer efficiency and are often deeply integrated with a single cloud provider. This leads to technical lock-in and makes it difficult to achieve flexible cross-cloud deployment and data collaboration.
The impossible trinity of performance, cost, and operations: High-performance parallel file systems offer high throughput and low latency but typically rely on all-flash arrays or dedicated hardware. This leads to high hardware investment and ongoing operational costs, plus complex deployment. Low-cost object storage offers good elasticity but is difficult to support the high-throughput I/O demands of GPU clusters in AI training scenarios. A common industry workaround is using a high-speed file system as a cache synchronized with S3. However, the extra data synchronization steps significantly reduce usability and fail to achieve efficient storage-compute synergy.
Difficulty in dataset version management: The rapid iteration cycle of robot models requires efficient and granular management of multiple dataset versions. Using physical copies for version control directly leads to exponentially higher underlying storage consumption, significantly increasing costs. Moreover, the difficulty of retrieving, reusing, and maintaining multi-version data also increases substantially.

Storage selection: JuiceFS vs. MinIO/S3 vs. PFS

To address these storage challenges, we established a clear evaluation framework for storage selection. A comprehensive comparative test was conducted on mainstream storage solutions across seven core dimensions: storage architecture, protocol compatibility, metadata performance, scalability, multi-cloud adaptability, cost efficiency, and operational complexity.

Comparison basis	JuiceFS	MinIO / Public Cloud S3	CephFS / Public Cloud FS (CPFS)
Storage architecture	Separation of metadata and data	Unified object storage	Metadata and data typically coupled, often with kernel-space parallel design
Protocol support	Full compatibility: POSIX, HDFS, S3 API, Kubernetes CSI	Primarily S3 API, with weak POSIX compatibility	POSIX-oriented; HDFS or S3 compatibility often requires plugins
Metadata performance	Very high: sub-millisecond latency, supports hundreds of billions of files per volume	Lower: high metadata overhead for massive small files; API call overhead about 10–30 ms	Medium to high: performance bottlenecks and complexity challenges at ultra-large scale (100M+ files)
Scalability	High: horizontal scaling, supports tens to hundreds of billions of files per volume	High: near-infinite storage capacity, but small-file management efficiency degrades with scale	Moderate: scaling limited by metadata nodes; operational complexity grows exponentially with scale
Multi-cloud adaptability	Native support	Relies on sync tools; cross-cloud data flow inefficient; global unified view difficult	Limited: often tightly bound to specific hardware or cloud provider; cross-cloud deployment is complex
Cost efficiency	High performance-to-cost ratio	Low (storage only): cheap storage, but low GPU utilization in high-throughput scenarios like AI training	High: often requires all-flash architecture or dedicated hardware; high operational labor cost

Based on the comparison results above, JuiceFS demonstrates significant advantages in core performance, scalability, multi-cloud adaptability, and cost efficiency. This makes it the preferred choice for our unified storage solution.
Furthermore, JuiceFS has been widely adopted in the autonomous driving industry. Leading companies such as Horizon Robotics have leveraged JuiceFS to manage data at the exabyte scale. This demonstrates its maturity and effectiveness in large-scale production environments.
For our specific application scenarios, JuiceFS' core technical advantages:

Decoupled architecture: JuiceFS adopts a metadata-data separation architecture, persisting data in cost-effective object storage (like S3 or OSS) while storing metadata in databases like Redis or TiKV. This decoupled design enables elastic storage scaling and reduces dependence on any single cloud provider.
Chunking and caching mechanisms: JuiceFS uses chunks, slices, and blocks to significantly improve small file read efficiency and enhance concurrent read/write performance. In addition, multi-level caching (memory, local SSD, distributed cache) reduces access latency for hot data. This meets the demands of high-throughput training workloads.
Cloud-native adaptability: By providing a CSI Driver, JuiceFS delivers persistent storage decoupled from compute nodes in Kubernetes environments, supporting stateless container deployment and cross-cloud migration. It enables data sharing, enhances application high availability and flexibility, and adapts to various Kubernetes deployment methods.
Full-stack support for AI training: JuiceFS fully supports POSIX, HDFS, and S3 API, and is compatible with mainstream AI frameworks such as PyTorch and TensorFlow. It can be integrated without code modifications, lowering the technical barrier for adoption.
Multi-cloud support: Its cross-cloud capabilities and high-performance metadata engine ensure efficient data flow, perfectly aligning with our strategy of "computing power on demand."

From a cost perspective, JuiceFS does not offer a significant cost advantage in the early stages of small-scale deployment. However, when data volume reaches the petabyte level—especially at the 10 PB or 100 PB scale—and is compared against all-flash storage solutions, its cost-efficient architecture built on object storage becomes fully evident. In addition, JuiceFS requires minimal operational overhead. Currently, we need only one engineer to manage the entire cloud platform and storage system, a fraction of the personnel required by traditional solutions.

From Community Edition to Enterprise Edition: addressing larger-scale scenarios

As our application continued to expand, we encountered limitations when using Redis as the metadata engine—specifically, physical memory capacity constrained data scalability. When the number of files approached the hundred-million level, metadata query latency increased significantly. This impacted the concurrency efficiency of training tasks. After using the clone feature, the metadata volume grew substantially. In addition, in cross-cloud scenarios, we faced higher demands for metadata synchronization and mirror file system capabilities. We also required more granular capacity controls and permission management at the directory level.

Considering these requirements—along with our desire to leverage local SSDs on GPU nodes to build a distributed cache layer for improved performance—we decided to deploy JuiceFS Enterprise Edition in parallel, migrating core scenarios such as ultra-large-scale directory management and multi-node collaborative training to this version. Through this scenario-based approach, we’ve effectively enhanced the adaptability of our overall storage system and established a solid foundation for future application growth. Below are the key features of the Enterprise Edition that we’ve applied in real-world scenarios.

High-performance metadata engine: solving the bottleneck of large-scale directory retrieval

For high-frequency operations such as traversing directories with hundreds of millions of files and deep pagination queries, we previously encountered the "slower as you query" problem with traditional storage solutions. When the number of files in a single directory exceeded 10 million, and the pagination offset surpassed 100,000 entries, response latency would spike from hundreds of milliseconds to several seconds. This severely impacted data filtering efficiency.

After switching to JuiceFS Enterprise Edition, its native tree-structured metadata storage architecture played a key role. Unlike the flat key-value storage used—which stores file metadata in a disordered manner—this tree structure allows direct navigation to directory levels, reducing the scope of metadata scans. In our actual tests, deep pagination queries (with an offset of 500,000 entries) in a directory containing 120 million files saw latency drop from 3.8 seconds to just 210 milliseconds. This fully met the retrieval needs of large-scale datasets. In addition, this engine supports storing hundreds of billions of files per volume, and we’ve already used it to manage three petabyte-scale training datasets stably, aligning with our application growth expectations.

Enterprise-grade distributed cache: improving data sharing efficiency in multi-node, multi-GPU training

In multi-node, multi-GPU training scenarios, we previously faced challenges such as low cache hit rates and cross-node bandwidth congestion. The open-source version only supports local caching on each node. This means that when multiple nodes pull the same dataset simultaneously, each node must access object storage independently. This resulted in single-node bandwidth utilization exceeding 90%, with average training job startup delays of up to 20 minutes.

With JuiceFS Enterprise Edition's distributed caching feature, we set up a distributed cache across a 12-node training cluster using just three commands. The dataset only needs to be pulled from object storage once and is cached in a pool built from local SSDs across the nodes. As a result, the cache hit rate for multi-node collaborative training increased from 45% to 92%, cross-node bandwidth utilization dropped to below 15%, and training job startup time was reduced to under three minutes. This significantly improved compute utilization.

Enhanced cross-cloud collaboration: building a low-operational-cost cross-cloud data foundation

Since our R&D environments are distributed across two cloud environments, we previously encountered challenges with slow cross-cloud data synchronization and high operational costs. Using traditional synchronization tools to maintain data consistency between the two clouds required configuring eight scheduled tasks, with an average synchronization delay of four hours, and dedicated personnel needed to investigate sync failures weekly.

By using the JuiceFS sync tool combined with our internal AI operations tools, we achieved automated configuration of synchronization policies. The system automatically adjusts sync priorities based on data heat levels, keeping cross-cloud data latency within 10 minutes. In addition, tasks such as failure retries and log alerts for synchronization are fully automated, eliminating the need for dedicated monitoring. This has reduced operational overhead by 70%, and we now stably support multiple training projects across two cloud platforms sharing the same dataset. Going forward, we plan to use the Enterprise Edition's mirror file system feature to further enhance cross-cloud data collaboration.

JuiceFS optimization

Client cache and write performance tuning

We need to pay attention to compatibility issues between caching strategies and Kubernetes resource limits. For example, using memory as a local cache path with improper configuration may lead to abnormal memory growth in the Mount Pod, or insufficient resource quota reservations may cause checkpoint loss or file handle write exceptions during long-running training tasks.

Regarding write performance tuning, enabling writeback mode can improve small file write throughput to some extent. However, considering production environment requirements for data consistency, we still adopt write-through synchronous mode to reduce data risks in extreme crash scenarios. It’s recommended to cautiously enable writeback mode only in scenarios with lower data reliability requirements, such as temporary computing or offline data cleaning, based on actual needs.

Deployment and network topology optimization

For more stable performance, it’s strongly recommended to deploy the metadata engine and compute nodes within the same region during deployment. In actual operations, we observed that cross-region deployment could increase metadata operation latency by several to ten times. This significantly impacted I/O-intensive operations such as data decompression. Deploying metadata services and GPU computing resources within the same region helps maintain performance while controlling network transmission costs, improving overall resource utilization efficiency.

Data warm-up and cache optimization

In a 10-gigabit network environment, fully utilizing JuiceFS' data warm-up and reasonably adjusting data block sizes based on application scenarios can better leverage network bandwidth capabilities and improve read throughput. Combined with the distributed cache architecture, this can effectively enhance data sharing efficiency in multi-node concurrent scenarios and improve cache hit rates during high-concurrency reads. This thereby optimizes the overall performance of large-scale AI training tasks.

Resource quotas and high availability guarantee

In enterprise-level multi-role operations and storage responsibility separation scenarios, to avoid operational risks caused by inconsistent configurations, it’s recommended to finely control resource quotas for JuiceFS CSI Driver in Kubernetes environments. By appropriately setting CPU and memory request/limit for Mount Pods, Pod restarts or node anomalies caused by resource preemption can be reduced. In practice, resource reservation ratios can be dynamically adjusted based on cluster load.

In addition, for scenarios with high application continuity requirements, the automatic mount point recovery feature for Mount Pods can be enabled to achieve automated fault recovery for storage services, further ensuring underlying storage stability.

Multi-tenancy

We provide independent file systems and storage buckets for large enterprise customers, while achieving isolation for small and medium-sized enterprises and end users through subdirectory-level directory isolation and permission control.

Large enterprises can flexibly scale throughput and capacity, avoiding performance bottlenecks associated with shared storage buckets. For small and medium-sized enterprises and end users, we ensure data security and independence through subdirectory isolation and permission control, while enabling accurate metering and billing.

This architecture ensures tenant isolation while flexibly allocating resources, improving system management efficiency.

Version management

Using the juicefs clone command, copies of original datasets can be quickly created and modified independently without affecting the source data. The clone operation only copies file metadata, while data only stores additional changes, saving underlying storage space. This feature supports managing multiple versions, facilitating rollback and recovery and ensuring data security and version control.

Summary

JuiceFS' characteristics in metadata performance, scalability, cross-cloud adaptability, and comprehensive cost efficiency have made it our choice for building a unified storage layer. Currently, we adopt both JuiceFS Community Edition and Enterprise Edition to accommodate different storage requirements across various application scenarios.

In the future, we plan to further implement JuiceFS in the embodied intelligence field, addressing specific storage needs in this scenario. These include high-throughput processing of time-series data, precise multi-modal data alignment, edge-cloud collaborative storage, and integrated management of simulation and real-world data.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

The Design Journey of FUSE: From Kernel-Space to User-Space File Systems

Sat, 14 Feb 2026 07:27:00 +0000

Filesystem in Userspace (FUSE) born in 2001, enables users to create custom file systems in user space. By lowering the barrier to file system development, FUSE empowers developers to innovate without modifying kernel code. JuiceFS, a high-performance distributed file system, leverages FUSE’s flexibility and extensibility to deliver robust storage solutions.

In this article, we’ll explore FUSE’s architecture and advantages, tracing the evolution of kernel file systems and network file systems that laid the groundwork for FUSE. Finally, we’ll share JuiceFS’ practical insights into optimizing FUSE performance for AI workloads. Since FUSE requires switching between user space and kernel space, it brings certain overhead and may lead to I/O latency. Therefore, many people have doubts about its performance. From our practical experience, FUSE can meet performance requirements in most AI scenarios. We’ll elaborate relevant details in this article.

Standalone file systems: Kernel space and VFS

The file system, as a core underlying component of the operating system, is responsible for frequent operations on storage devices. It was initially designed entirely in kernel space. The kernel concept emerged as computer hardware and software became increasingly complex, and operating systems separated the code for managing underlying resources from user programs.

Kernel space and user space

Kernel space:

The kernel is code with super privileges. It manages a computer’s core resources, such as CPU, memory, storage, and network.
When kernel code runs, the program enters kernel space, enabling full access to and control over underlying hardware. Due to the kernel’s high privileges, its code must undergo stringent testing and verification, and ordinary users cannot modify it freely.

User space:

It’s the code of various applications we commonly use, such as browsers and games.
In user space, the permissions of programs are strictly limited and cannot directly access important underlying resources.

image: Kernel space and user space
caption: Kernel space and user space

If an application needs to use a file system, it must access it through the interface designed by the operating system, such as the commonly used OPEN, READ, and WRITE, which are system calls. The role of system calls is to build a bridge between user space and kernel space. Modern operating systems often define hundreds of system calls, each with its own clear name, number, and parameters.

When an application makes a system call, it enters a section of kernel space code and returns the results to user space after execution. It’s worth noting that this entire process from user space to kernel space and then back to user space belongs to the same process category.

image: System call architecture
caption: System call architecture

Virtual file systems

After understanding the background knowledge above, we’ll briefly explain how user space and kernel space interact when a user calls a file system interface.

The kernel encapsulates a set of universal virtual file system interfaces through virtual file system (VFS), exposes them to user space via system calls, and provides programming interfaces to the underlying file systems. The underlying file systems need to implement their own file system interfaces according to the VFS format.

The standard process for user space to access the underlying file system: a system call -> the VFS -> the underlying file system -> the physical device

For example, when we call open in an application, it carries a path as its parameter. After this call reaches the VFS layer, the VFS searches level by level in its tree structure based on this path. Ultimately, it finds a corresponding target and its affiliated underlying file system. This underlying file system also has its own implementation of the open method. Then, it passes this open call to the underlying file system.

The Linux kernel supports dozens of different file systems. For different storage media such as memory or networks, different file systems are used for management. The most critical point is that the extensibility of VFS enables the Linux system to easily support a variety of file systems to meet various complex storage needs; at the same time, this extensibility also provides a foundation for FUSE to implement kernel-space functions in user space later.

Network file systems: Breaking kernel boundaries

With the growth of computing needs, the performance of a single computer gradually could not meet the increasing computing and storage requirements. People began to introduce multiple computers to share the load and improve overall efficiency.

In this scenario, an application often needs to access data distributed on multiple computers. To solve this problem, people proposed the concept of introducing a virtual storage layer in the network, virtually mounting the remote computer's file system (such as a certain directory) through a network interface to the local computer's node. The purpose of this is to enable the local computer to seamlessly access remote computer data as if the data was stored locally.

image: Network file system (NFS) architecture
caption: Network file system (NFS) architecture

Specifically, if a local computer needs to access remote data, a subdirectory of the remote computer can be virtually mounted to a node of the local computer through a network interface. In this process, the application does not need to make any modifications and can still access these paths through standard file system interfaces as if they were local data.

When the application performs operations on these network paths (such as hierarchical directory lookup), these operations are converted into network requests and sent to the remote computer in the form of remote procedure call (RPC) for execution. After receiving these requests, the remote computer performs corresponding operations (such as finding files and reading data) and returns the results to the local computer.

The process above is a simple implementation of the NFS protocol. As a network file system protocol, NFS provides an efficient solution for resource sharing between multiple computers. It allows users to mount and access remote file systems as conveniently as operating local file systems.

Traditional file systems typically run entirely in the kernel space of a single node, while NFS was the first to break this limitation. The server-side implementation combines kernel space with user space. The subsequent design of FUSE was inspired by this approach.

FUSE: File system innovation from kernel to user space

With the continuous development of computer technology, many emerging application scenarios require using custom file systems. Traditional kernel-space file systems have high implementation difficulties and version compatibility issues. The NFS architecture first broke through the limitations of the kernel.

Based on this, someone proposed an idea: Can we transplant the NFS network protocol to a single node, transfer the server-side functionality to a user-space process, while retaining the client running in the kernel, and use system calls instead of network communication to realize file system functions in user space? This idea eventually led to the birth of FUSE.

In 2001, Hungarian computer scientist Miklos Szeredi introduced FUSE, a framework that allows developers to implement file systems in user space. The core of FUSE is divided into two parts: the kernel module and the user-space library (libfuse).

Its kernel module, as part of the operating system kernel, interacts with VFS, forwarding file system requests from VFS to user space, and returning the processing results of user space to VFS. This design allows FUSE to implement custom file system functions without modifying kernel code.

The FUSE user-space library (libfuse) provides an API library that interacts with the FUSE kernel module and helps users implement a daemon running in user space. The daemon handles file system requests from the kernel and implements specific file system logic.

image: FUSE workflow
caption: FUSE workflow

In specific implementations, the user-space daemon and kernel module collaborate through the following steps to complete file operations:

Request reception

1.1 The kernel module registers a * character device (/dev/fuse) as a communication channel. The daemon reads requests from this device by calling read().

1.2 If the FUSE request queue of the kernel is empty, read() enters a blocking state. At this time, the daemon pauses execution and releases CPU until a new request appears in the queue (implemented through the kernel's wait queue mechanism).

1.3 When an application initiates a file operation (such as open and read), the kernel module encapsulates the request into a specially formatted data packet and inserts it into the request queue, waking the blocked daemon.

Request processing
After the daemon reads the request data packet from the character device, it calls the corresponding user-space processing function according to the operation type (such as reading, writing, and creating a file).
Result returning
After processing is complete, the daemon serializes the result (such as the content of the read file or error code) according to the FUSE protocol format and writes the data packet back to the character device through write().

After the kernel module receives the response:

3.1 It parses the data packet content and passes the results to the waiting application.

3.2 It wakes up the system call blocked in the application to continue executing subsequent logic.

The emergence of FUSE brought revolutionary changes to file system development. By migrating the implementation of file systems from kernel space to user space, FUSE significantly reduced development difficulty, improved system flexibility and extensibility, and was widely applied in various scenarios such as network file systems, encrypted file systems, and virtual file systems.

JuiceFS: A FUSE user-space distributed file system

In 2017, with the full entry of IT infrastructure into the cloud era, the architecture faced unprecedented challenges. In this background, JuiceFS was born. As a distributed file system based on object storage, it uses FUSE technology to build its file system architecture, using FUSE’s flexible extensibility to meet the diverse needs of cloud computing environments.

Through FUSE, the JuiceFS file system can be mounted to servers in a POSIX-compatible manner. It treats massive cloud storage as local storage. Common file system commands, such as ls, cp, and mkdir, can be used to manage files and directories in JuiceFS.

image: JuiceFS 架构图（第四版）-第 2 页-winfsp (2)
caption: JuiceFS Community Edition architecture

Let’s take a user mounting JuiceFS and then opening one of its files as an example. The request first goes through the kernel VFS, then is passed to the kernel's FUSE module, and communicates with the JuiceFS client process through /dev/fuse device. The relationship between VFS and FUSE can be simply regarded as a client-server protocol, with VFS acting as the client requesting service, and the user-space JuiceFS acting as the server role, handling these requests.

The workflow is as follows:

After JuiceFS is mounted, the go-fuse module inside JuiceFS opens /dev/fuse to obtain mount fd and start several threads to read the FUSE requests of the kernel.
The user calls the open function, enters the VFS layer through the C library and system call, and the VFS layer transfers the request to the kernel's FUSE module.
The kernel FUSE module puts the open request into the queue corresponding to the fd of juicefs mount according to the protocol and wakes up the read request thread of go-fuse to wait for the processing result.
The user-space go-fuse module reads the FUSE request and calls the corresponding implementation of JuiceFS after parsing the request.
go-fuse writes the processing result of this request into mount fd, that is, into the FUSE result queue, and then wakes up the application waiting thread.
The application thread is awakened, gets the processing result of this request, and then returns to the upper layer.

Due to the frequent switching between user space and kernel space required by FUSE, many people have doubts about its performance. In fact, this is not entirely the case. We conducted a set of tests using JuiceFS.

The testing environment: 1.5 TB RAM, an Intel Xeon 176-core machine, a 512 GB sparse file on JuiceFS

We used fio to perform sequential read tests on it:

Mount parameters: ./cmd/mount/mount mount --no-update --conf-dir=/root/jfs/deploy/docker --cache-dir /tmp/jfsCache0 --enable-xattr --enable-acl -o allow_other test-volume /tmp/jfs
The fio command: fio --name=seq_read --filename=/tmp/jfs/holefile --rw=read --bs=4M --numjobs=1 --runtime=60 --time_based --group_reporting

This excluded the constraints of hardware disks and tested the limit bandwidth of the FUSE file system.

image: all-memory
caption: JuiceFS sequential read throughput in all-memory

Testing results:

The single-thread bandwidth under a single mount point reached 2.4 GiB/s.
As the number of threads increased, the bandwidth could grow linearly. At 20 threads, it reached 25.1 GiB/s. This throughput already meets most actual application scenarios.

In terms of the use of FUSE, JuiceFS has implemented the smooth upgrade feature. By ensuring the consistency of mount fd, users can upgrade the JuiceFS version or modify the mount parameters without re-mounting the file system and interrupting the application. For details, see Smooth Upgrade: Implementation and Usage.

FUSE also has some limitations. For example, processes accessing the FUSE device require high permissions, especially in container environments, usually requiring privileged mode to be enabled. In addition, containers are usually transient and stateless. If a container exits unexpectedly and data is not written to disk in time, there is a risk of data loss.

Therefore, for Kubernetes scenarios, the JuiceFS CSI Driver allows applications to access the JuiceFS file system with non-privileged containers. The CSI driver manages the lifecycle of the FUSE process to ensure that data can be written to disk in time and will not be lost. For details, see K8s Data Persistence: Getting Started with JuiceFS CSI Driver.

Conclusion

FUSE decouples user space from kernel space, providing developers with great flexibility and convenience in implementing file systems in user space. Especially in modern computing environments such as cloud computing and distributed storage, FUSE makes building and maintaining complex storage systems more efficient, customizable, and easy to expand.

JuiceFS is based on FUSE and implements a high-performance distributed file system in user space. In the future, we’ll continue exploring optimization methods for FUSE and continuously improving the performance and reliability of file systems to meet increasingly complex storage needs and provide users with stronger data management capabilities.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

JuiceFS Enterprise 5.3: 500B+ Files per File System & RDMA Support

Wed, 04 Feb 2026 02:59:53 +0000

JuiceFS Enterprise Edition 5.3 has recently been released, achieving a milestone breakthrough by supporting over 500 billion files in a single file system. This upgrade includes several key optimizations to the metadata multi-zone architecture and introduces remote direct memory access (RDMA) technology for the first time to enhance distributed caching efficiency. In addition, version 5.3 enhances write support for mirrors and provides data caching for objects imported across buckets. It aims to support high-performance requirements and multi-cloud application scenarios.

JuiceFS Enterprise Edition is designed for high-performance scenarios. Since 2019, it has been applied in machine learning and has become one of the core infrastructures in the AI industry. Its customers include large language model (LLM) companies such as MiniMax and StepFun; AI infrastructure and applications like fal and HeyGen; autonomous driving companies like Momenta and Horizon Robotics; and numerous leading technology enterprises across various industries leveraging AI.

Single file system supports 500 billion+ files

The multi-zone architecture is one of JuiceFS' key technologies for handling hundreds of billions of files, ensuring high scalability and concurrent processing capabilities. To meet the growing demands of scenarios like autonomous driving, version 5.3 introduces in-depth optimizations to the multi-zone architecture, increasing the zone limit to 1,024 and enabling a single file system to store and access at least 500 billion files (each zone can store 500 million files, with a maximum of 2 billion).

The figure below shows JuiceFS Enterprise Edition architecture, with a single zone in the lower left corner:

image: 企业版架构
caption: JuiceFS Enterprise Edition

This breakthrough presents exponentially increasing challenges in system performance, data consistency, and stability, backed by a series of complex underlying optimizations and R&D efforts.

Cross-zone hotspot balancing: automated monitoring and hotspot migration, with manual ops tools

In distributed systems, hotspots are a common challenge. Especially when data is distributed across multiple zones, some zones may experience higher loads than others. This leads to imbalance that impacts system performance.

When the number of zones reaches hundreds, hotspot issues become more prevalent. Particularly with smaller datasets and larger numbers of files, read/write hotspots exacerbate latency fluctuations.

We introduced an automated hotspot migration mechanism to move frequently accessed files to other zones, distributing the load and reducing pressure on specific zones. However, in practice, relying solely on automated migration cannot fully resolve all issues. In certain special or extreme scenarios, automated tools may not respond promptly. Therefore, alongside automated monitoring and migration, we added manual operational tools, allowing administrators to intervene in complex scenarios, perform manual analysis, and implement optimization solutions.

Large-scale migration: improved migration speed, small-batch concurrent migration

Facing zones with excessive hotspots, early migration operations were simple. However, as the system scale expanded, migration efficiency gradually decreased. To address this, we introduced a small-batch concurrent migration strategy, breaking down high-access directories into smaller chunks and migrating them in parallel to multiple lower-load zones. This quickly scatters hotspots and restores normal application access.

Enhanced reliability self-checks: automatic repair and cleanup of intermediate migration states

In large-scale clusters, the probability of distributed transaction failures increases significantly, especially during extensive migration processes. To address this, we enhanced reliability detection mechanisms, adding periodic background checks to scan cross-zone file states, particularly focusing on intermediate state issues, and automatically performing repairs and cleanup.

Previously, the system encountered issues with residual intermediate state data. While these did not affect operations in the short term, over time they could lead to errors. Through enhanced self-check mechanisms, we ensure the background periodically scans and promptly handles intermediate state issues, improving system stability and reliability.

Beyond the three key optimizations above, we also made multiple improvements to the console to better adapt to managing more zones. We optimized concurrent processing, operational tasks, and query displays, enhancing overall performance and user experience. Specifically, we refined UI design to better showcase system states in large-scale zone environments.

Performance stress test for hundreds of billions of files

We conducted large-scale tests using a custom mdtest tool on Google Cloud, deploying 60 nodes, each with over 1 TB of memory. In terms of software configuration, we increased the number of zones to 1,024. The deployment method was similar to previous setups, but to reduce memory consumption, we deployed only one service process, with two others as cold backups.

image: 压测
caption: JuiceFS Enterprise Edition 5.3 test

JuiceFS Enterprise Edition 5.3 test:

Test duration: Approximately 20 hours
Total files written: About 400 billion files
Write speed: 5 million files per second
Memory usage: About 35% to 40%
Disk usage: 40% to 50%, primarily for metadata persistence, with good utilization

Based on our experience, if using a configuration with one service process, one hot backup, and one cold backup, memory usage increases by 20% to 30%.

Due to limited cloud resources, this test only wrote up to 400 billion files. During stress testing, the system performed stably, with hardware resources still remaining. We’ll continue to attempt larger-scale tests in the future.

Support for RDMA: increased bandwidth cap, reduced CPU usage

This new version introduces support for RDMA technology for the first time. Its basic architecture is shown in the diagram below. RDMA allows direct access to remote node memory, bypassing the operating system's network protocol stack. This significantly improves data transfer efficiency.

image: rdma
caption: RDMA principle architecture

The main advantages of RDMA include:

Low latency: By enabling direct memory-to-memory transfers and bypassing the OS network protocol layers, it reduces CPU interrupts and context switches. This lowers latency.
High throughput: RDMA uses hardware for direct data transfer, better utilizing the bandwidth of network interface cards (NICs).
Reduced CPU usage: In RDMA, data copying is almost entirely handled by the NIC, with the CPU only processing control messages. This allows the NIC to handle hardware transfers, freeing up CPU resources.

In JuiceFS, network request messages between clients and metadata services are small, and existing TCP configurations already meet the needs. However, in distributed caching, file data is transferred between clients and cache nodes. Using RDMA can effectively improve transfer efficiency and reduce CPU consumption.

image: 网络对比
caption: CPU usage comparison: TCP vs. RDMA

We conducted 1 MB random read tests using 160 Gbps NICs, comparing versions 5.1, 5.2 (using TCP networking) with version 5.3 (RDMA), and observed CPU usage.

Tests showed that RDMA effectively reduces CPU usage:

In version 5.2, CPU usage was nearly 50%.
In version 5.3, with RDMA optimization, CPU usage dropped to about one-third. Client and cache node CPU usage decreased to 8 cores and 5 cores respectively, with bandwidth reaching 20 GiB/s.

In previous tests, we found that while TCP ran stably on 200G NICs, fully saturating bandwidth was challenging, typically achieving only 85%-90% utilization. For customers requiring higher bandwidth (such as 400G NICs), TCP could not meet demands. However, RDMA can more easily reach hardware bandwidth limits, providing better transfer efficiency.

If customers have RDMA-capable hardware and high bandwidth requirements (for example, NICs greater than 100G) and wish to reduce CPU usage, RDMA is a technology worth trying. Currently, our RDMA feature is in public testing and has not yet been widely deployed in production environments.

Enhanced write support for mirrors

Initially, mirror clusters were primarily used for read-only mirroring in enterprise products. As users requested capabilities like writing temporary files (such as training data) in mirrors, we provided write support for mirrors.

image: 企业版镜像系统
caption: JuiceFS Enterprise Edition’s mirror file system architecture

The mirror client implements a read-write separation mechanism. When reading data, the client prioritizes fetching from the mirror cluster to reduce latency. When writing data, it still writes to the source cluster to ensure data consistency. By recording and comparing metadata version numbers, we ensure strong consistency between the mirror client and source cluster client views of the data.

To improve availability, version 5.3 introduces a fallback mechanism. When the mirror becomes unavailable, client read requests automatically fall back to the source cluster. This ensures application continuity and avoids interruptions caused by mirror cluster failures. We also optimized deployments in multi-mirror environments. Previously, the mirror end required two hot backup nodes to ensure high availability. Now, with the improved fallback feature, deploying a single mirror node can achieve similar effects. This ensures application continuity and reduces costs, especially beneficial for users requiring multiple mirrors.

Through this improvement, we not only reduced hardware costs but also found a balance between high availability and low cost. For users deploying mirrors in multiple locations, reducing metadata replicas further lowers overall costs.

Simplified operations & increased flexibility: providing cross-bucket data cache for imported objects

In JuiceFS, users can use the import command to bring existing files from object storage under unified management. This is convenient for users already storing large amounts of data (for example, tens of petabytes). However, in previous versions, this feature only supported caching for objects within the same data bucket. This meant imported objects had to reside in the same bucket as the existing file system data. This limitation had certain practical constraints.

In version 5.3, we improved this feature. Users can now provide caching capability for any imported objects, regardless of whether they come from the same data bucket. This allows users more flexibility in managing objects across different data buckets, avoiding strict bucket restrictions and enhancing data management freedom.

In addition, previously, if users had data distributed across multiple buckets and wanted to provide caching for that data, they needed to create a new file system for each bucket. In version 5.3, users only need to create one file system (volume) to uniformly manage data from multiple buckets and provide caching for all buckets.

Other important optimizations

Trace feature

We added the trace feature, a feature provided by the Go language itself. Through this, advanced users can perform tracing and performance analysis, gaining more information to help quickly locate issues.

Trash recovery

In previous versions, especially with multiple zones, sometimes the paths recorded in the trash were incomplete. This led to anomalies during recovery, where files were not restored to the expected locations. To address this, in version 5.3, when deleting files, we record the original file path, ensuring more reliable recovery capabilities.

Python SDK improvements

In earlier versions, we released the Python SDK, providing basic read/write functionalities for Python users to interface with our system. In version 5.3, we not only strengthened basic read/write functions but also added support for operational subcommands. For example, users can directly call commands like juicefs info or warmup via the SDK without relying on external system commands. This simplifies coding efforts and avoids potential performance bottlenecks from frequently calling external commands.

The Windows client

We previously launched a beta version of the Windows client and have received some user feedback. After improvements, the current version shows significant enhancements in mount reliability, performance, and compatibility with Linux systems. In the future, we plan to further refine the Windows client, providing an experience closer to Linux for users reliant on Windows.

Summary

Compared to expensive dedicated hardware, JuiceFS helps users balance performance and cost when addressing data growth by flexibly utilizing cloud or existing customer storage resources. In version 5.3, by optimizing the metadata zone architecture, a single file system can support over 500 billion files. The first-time introduction of RDMA technology significantly improves distributed caching bandwidth and data access efficiency, reduces CPU usage, and further optimizes system performance. In addition, we enhanced features like write support for mirrors and caching, improving the performance and operational efficiency of large-scale clusters and optimizing user experience.

Cloud service users can now directly experience JuiceFS Enterprise Edition 5.3 online, while on-premises deployment users can obtain upgrade support through official channels. We’ll continue to focus on high-performance storage solutions, partnering with enterprises to tackle challenges brought by continuous data growth.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

How Just Two Cache Nodes Achieved 1.45 TB/s Throughput

Thu, 29 Jan 2026 12:45:00 +0000

As application demands for large-scale concurrent reads and data distribution increase, such as in film and television rendering scenarios, traditional storage solutions like NAS often require significant investment in additional cache resources when the number of concurrent clients grows. To improve response times, data warm-up is also typically necessary. This not only incurs extra time overhead but also further increases resource strain.

JuiceFS, a distributed file system built on object storage, uses its high-performance architecture to aggregate throughput and reduce latency via distributed caching. It provides efficient support for large-scale concurrent client reads.

In this article, we’ll share a recent real-world test case, demonstrating how we successfully aggregated 1.45 TB/s of bandwidth using 4,000 application nodes and, in the process, ensured system stability by introducing a two-level cache pool configured with only two independent cache nodes.

Through this article, we aim to provide a practical solution for storage bottlenecks in high-concurrency, high-throughput scenarios and hope it sparks further discussion and exploration of storage optimization methods.

Traditional NAS: more nodes, slower storage

Our customer in this case was a film and television render farm user, where thousands of Windows nodes are launched simultaneously for daily rendering jobs.

Application characteristics: Each node needs to read a batch of files (mostly reusable assets) to its local storage during rendering.
Original pain points: When using the original public cloud NAS storage (mounted via SMB), increasing the number of nodes forced continuous addition of backend SMB service nodes to handle the surge in traffic and IOPS. This led to a steep rise in management complexity and cost. When concurrent nodes exceeded 1,000, the storage system often became overwhelmed.
Critical requirement: An urgent need for a capability beyond the storage foundation to shoulder the pressure of application throughput.

image: Multi-node NAS architecture
caption: Multi-node NAS architecture

In the SMB service model, each client reads the full data volume from the SMB storage. This results in sustained high traffic at the server side. Administrators must continuously monitor the health of the SMB services. Once the load approaches maximum capacity, they need to promptly scale out, providing storage capability that matches the cluster size. This significantly increases operational pressure.

Utilizing idle resources: using numerous application nodes as distributed cache

JuiceFS released a new Windows client after Community Edition 1.3 and Enterprise Edition 5.2. It supports the mounting of the file system as a local drive via a .exe process, with usage similar to Linux.

However, in scenarios with massive numbers of clients, simply switching the application to the standard JuiceFS mount point -> distributed cache -> object storage chain could concentrate traffic on the independent cache layer. This potentially creates a new performance bottleneck. Instead of continuously scaling dedicated cache nodes, a shift in perspective is more effective: use the idle bandwidth and disk space of the vast number of application nodes, pooling them into a massive distributed cache pool.

JuiceFS distributed cache mode (P2P mode): A file only needs to be read once within the cluster; subsequent requests from other nodes fetch it directly from the neighboring nodes in the P2P cache pool.
Object storage side: The back-to-source traffic is extremely low. After the initial cold read of a file, subsequent traffic is almost entirely handled by the cache pool.
Resource requirements: No dedicated cache hardware is needed, only requiring each application node to contribute a portion of its disk and bandwidth.

In this solution, we did not configure a single independent cache node. All application nodes act as both consumers and providers (P2P mode).

image: JuiceFS distributed cache
caption: JuiceFS distributed cache deployment architecture

Case study: 4,000 app nodes aggregate to 1.45 TB/s throughput

Test task: Each Windows node, without any warm-up (cold read), read 16 large files of 2 GB each. The total time for all nodes to finish reading was measured, observing the variance in time per node and checking for long-tail effects.

Configuration strategy: The 4,000 nodes were divided into multiple subgroups (500 nodes per group). The 16 data blocks of 2 GB each were distributed across nodes within a group using hashing to avoid all nodes simultaneously requesting data from the object storage. This would cause congestion.

The cold read process for the JuiceFS client:

The Windows client read the 16 files of 2 GB each. Using a consistent hashing topology, it located the corresponding nodes for these data blocks within its 500-node cache group and sent requests to them.
Upon receiving a data block request, the cache node found a local cache miss (cold read), so it fetched the data block from the object storage. After retrieval, it returned the data to the client and wrote it to its local cache for reuse by subsequent requests.
Once the client had retrieved complete data blocks from all cache service nodes, the test ended. The time distribution (maximum, minimum) for all clients to finish reading was compiled to evaluate variance and long-tail situations.

The results for reading this batch of 16*2 GB files with different numbers of client nodes are as follows:

Number of clients	Peak aggregated throughput	Total time (range/average)
2,000	729 GB/s	92s ~ 136s / Avg 107s
2,500	921 GB/s	87s ~ 109s / Avg 98s
3,000	1.11 TB/s	93s ~ 121s / Avg 106s
3,500	1.34 TB/s	89s ~ 112s / Avg 100s
4,000	1.45 TB/s	92s ~ 115s / Avg 101s

The figure below shows aggregated throughput performance of JuiceFS with different client counts:

image: I:O aggregation performance
caption: I/O aggregation performance

The results met expectations in all aspects:

Stability: Whether with 2,000 or 4,000 nodes, the total time to read the data remained stable at around 100 seconds.
Scalability: The 4,000 nodes successfully aggregated an ultra-high bandwidth of 1.45 TB/s. Theoretically, within the limits of metadata capacity, this architecture can achieve continuous horizontal scaling, potentially supporting cache node clusters at the scale of tens of thousands.

Reference mount parameters for application nodes:

juicefs.exe mount juice-fs X: --cache-group=primary --buffer-size=4096 --enable-kernel-cache --as-root --subgroups=8

Thus, without using a single independent cache service node, we aggregated 1.45 TB/s of read capability using the customer’s application nodes. By offloading the vast majority of traffic from the object storage to this distributed cache layer formed by client nodes, we alleviated the burden on the underlying storage at zero additional hardware cost.

In actual application scenarios, such extremely high throughput might not always be achievable, as cache efficiency is often related to data duplication. In practice, the files read by each node are not entirely the same. Nonetheless, this solution is an effective method for storage scaling. Even a partial improvement can yield significant benefits with almost no additional cost.

Enhanced stability: two-level distributed caching

While the caching effect was impressive, the customer expressed concerns about system stability. For example, in some scenarios, application nodes might be destroyed immediately after completing their tasks, and these application nodes also served as cache nodes. When a large number of application nodes went offline suddenly, the cache stored on these nodes was also lost. This could cause a massive surge of traffic to fall back to the object storage, turning it into a bottleneck and affecting overall stability. To address caching performance issues caused by application node volatility, we proposed a two-level distributed caching solution, as shown in the architecture diagram below:

image: JuiceFS two-level
caption: JuiceFS two-level cache architecture

The second-level (L2) cache pool sits between the first-level (L1) cache pool and the object storage. On an L1 cache miss, data is first attempted to be retrieved from L2. If it's also a miss in L2, it falls back to the object storage. This effectively mitigates the impact of L1 cache node churn. Since L2 only handles the fallback traffic from L1 misses (including cold reads and warm-up), its capacity and performance planning only need to cover the available throughput of the object storage side. In this test, configuring just two independent cache nodes as L2 was sufficient to meet the demand.

With the addition of the L2 cache pool, the read process changes as follows:

The Windows client reads the 16 files of 2 GB each. It determines the specific node for the data block within its L1 cache group via the consistent hashing topology and simultaneously requests data from the L2 cache group.
Due to a cold read, there is no data in the L2 cache group. The L2 cache node fetches it from the object storage and fills the L2 cache pool.
The data block is returned from the L2 cache pool to the L1 cache pool for population, and then distributed P2P within the L1 cache pool.
At this point, most traffic is concentrated within the L1 cache pool. L2 only handles the minimal traffic falling back to object storage. Therefore, even with only a few L2 nodes, they do not become a performance bottleneck. The role of the L2 cache pool is to act as a low-latency local substitute for the object storage.

Even if a large number of L1 application nodes go offline, causing the cache topology to change and requiring data blocks to be re-downloaded. Data can still be fetched from the nearby L2 cache pool. As long as the proportion of nodes going offline is controlled reasonably, application operations are barely affected.

L2 cache group mount parameters:

juicefs mount juice-fs /jfs-cache --cache-group=secondary --cache-size=-1 --cache-dir=/data* --free-space-ratio=0.01 --buffer-size=10240 --max-downloads=400

L1 (application node) mount parameters:

juicefs.exe mount juice-fs X: --cache-group=primary --second-group=secondary --buffer-size=4096 --enable-kernel-cache --as-root --subgroups=8

Furthermore, the two-level distributed caching is highly suitable for scenarios requiring the reuse of existing cache pools.

For example, consider a batch of cache-pool applications located in Seattle, with a total capacity of 2 PiB, named cache-group-st. Suddenly, applications in Chicago also need to use the same data, which is almost the same as the Seattle data.

Instead of warming up the 2 PiB of data from Seattle’s object storage, we can configure the Chicago cache group with --second-group=cache-group-st. When Chicago application requests data, it prioritizes reading from the Seattle cache pool over a dedicated line, achieving very fast and stable speeds (within 2 ms latency). This eliminates the complex process of repeated data warm-up, allowing the Chicago applications to launch directly. This is extremely convenient.

Summary

Through this extreme stress test with 4,000 nodes, we successfully transformed the large-scale idle resources of a compute cluster into a storage pool with up to 1.45 TB/s of throughput. The introduction of a secondary cache effectively addressed "last-mile" stability concerns. By employing JuiceFS' storage software architecture, the potential of client clusters can be fully unlocked, achieving significant performance improvements without increasing additional hardware costs.

This solution is applicable to scenarios such as:

High-concurrency repeated read scenarios: Such as model training/inference data fetching, container image distribution, and film and television rendering. The more nodes, the greater the P2P cache benefit.
Elastic computing scenarios: Where application nodes frequently scale in and out on a large scale (such as spot instances). Using a two-level cache architecture ensures continuity and stability of data access.
Hybrid cloud / multi-cloud architectures: Leveraging the secondary caching mechanism allows for the reuse of cache pool resources across different regions, minimizing object storage calls and transfer costs associated with repeated warm-up.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

Juicedata Joins the Agentic AI Foundation as a Silver Member

Wed, 21 Jan 2026 03:16:00 +0000

Juicedata is pleased to announce that we have joined the Agentic AI Foundation (AAIF) as a Silver Member. We are excited to collaborate with a growing global community to advance open, interoperable, and production-ready foundations for the agentic AI era.

What is AAIF?

AAIF is a Linux Foundation–hosted initiative created to provide vendor-neutral governance and a shared home for collaboration on open standards, protocols, and projects that enable agentic AI systems to work reliably across environments and vendors.

AAIF is anchored by several widely discussed building blocks for agentic systems, including:

Model Context Protocol (MCP) – a standard approach for connecting models/agents to tools and context in a consistent way
goose – an open project focused on agentic runtime workflows
AGENTS.md – a lightweight, practical standard intended to improve how agents interact with codebases and developer workflows

Why Juicedata joined AAIF: the file system is critical infrastructure

Agentic AI shifts systems from “single prompt → single response” into continuous, tool-using, multi-step execution. That evolution raises the bar for the data layer. In practice, nearly every production-grade agentic system depends on a modern AI data pipeline that must handle:

Data processing and feature generation: massive parallel reads/writes, high metadata churn, and mixed file sizes
Pre-training: high-throughput sequential scans, streaming datasets, and large-scale checkpointing
Post-training (alignment, SFT, RLHF/RLAIF): frequent dataset versioning, sampling, and experiment tracking
Inference and model distribution: fast model artifact delivery, cold-start mitigation, and predictable loading
Multi-region deployment: mirrored datasets and model artifacts, replication workflows, and consistency guarantees across regions

In other words: the file system is on the critical path for agentic applications and modern AI pipelines. If storage is slow, inconsistent, or operationally fragile, the entire agentic stack becomes unreliable—regardless of how good the model or agent framework is.

Juicedata joined AAIF because we believe the agentic ecosystem needs not only protocols and agent runtimes, but also a robust, open, and scalable data foundation that works across clouds, regions, and heterogeneous compute.

Juicedata and JuiceFS: built for modern AI data pipelines

Juicedata builds infrastructure software for data-intensive workloads. Our flagship product, JuiceFS, is a cloud-native distributed file system designed to provide a unified namespace and high-performance access for massive-scale datasets across hybrid and multi-cloud environments.

JuiceFS is commonly adopted where teams need:

High-throughput read/write performance with predictable latency
Strong metadata capabilities for billions of files and high-concurrency workloads
Elastic scale with object storage economics and cloud-native operations
Operational simplicity for heterogeneous compute (Kubernetes clusters, GPU fleets, autoscaling inference, etc.)
Data mobility across regions and clouds, including replication and mirrored distribution patterns

These capabilities map directly to the needs of agentic systems, which increasingly behave like always-on dataflow engines: continuously reading context, writing artifacts, caching intermediate results, and distributing models and datasets across dynamic compute.

Where JuiceFS is used today

We are proud that JuiceFS supports a broad range of AI and data-intensive scenarios in production. Organizations we work with include:

Foundation Models: Zhipu GLM, MiniMax
AI Applications: HeyGen, fal.ai, Loveart, Gensmo, RunComfy, PixVerse
NeoCloud and MaaS: Baseten, Cerebrium, GMICloud
Autonomous Driving: Momenta, Horizontal Robotics

This diversity matters. Agentic systems are not confined to one “AI app” shape—foundation model builders, AI-native product teams, MaaS platforms, and autonomous driving pipelines all face different operational constraints, but they share a common requirement: fast, dependable data access at scale.

How we plan to contribute to AAIF

By joining AAIF, Juicedata aims to be an active and constructive participant in the community. Specifically, we intend to collaborate on:

Reference architectures for agentic AI data foundations: best practices for dataset layout, caching strategy, and multi-region artifact distribution
Operational patterns for large-scale agentic deployments: reliability, observability, and performance tuning of the storage layer under agent-driven workloads
Ecosystem integrations: improving the “plumbing” between agentic tooling (including emerging standards like MCP) and the data layer that agents depend on
Community knowledge-sharing: publishing benchmark methodologies, lessons learned, and production playbooks drawn from real-world workloads

AAIF’s emphasis on open governance and interoperability aligns with Juicedata’s belief that the agentic era will be won by ecosystems—not silos.

Looking ahead

Agentic AI is moving quickly from experimentation to production. As that happens, infrastructure decisions that once looked “implementation-specific” become strategic. Data layout, replication, cache coherence, and model distribution are no longer secondary concerns—they are core determinants of product reliability and user experience.

Juicedata joined AAIF to work closely with the global community and help ensure that the data foundation for agentic systems is open, scalable, and production-grade.

We look forward to collaborating with fellow members and contributors—and to building the agentic AI era in the open.

From GlusterFS to JuiceFS: Lightillusions Achieved 2.5x Faster 3D AIGC Data Processing

Thu, 08 Jan 2026 10:09:00 +0000

Lightillusions is a company specializing in spatial intelligence technology, integrating 3D vision, computer graphics, and generative models to build innovative 3D foundation models. Our company is led by Ping Tan, a professor at the Hong Kong University of Science and Technology (HKUST) and Director of the HKUST-BYD Joint Laboratory.

Unlike 2D models, a single 3D model can be several gigabytes in size, especially complex models like point clouds. When our data volume reached petabyte scales, management and storage became significant challenges. After trying solutions like NFS and GlusterFS, we chose JuiceFS, an open-source high-performance distributed file system, to build a unified storage platform. This platform now serves multiple scenarios, supports cross-platform access including Windows and Linux, manages hundreds of millions of files, improves data processing speed by 200%–250%, enables efficient storage scaling, and greatly simplifies operations and maintenance. This allows us to focus more on core research.

In this article, we’ll break down the unique storage demands of 3D AIGC, share why we selected JuiceFS over CephFS, and walk through the architecture of our JuiceFS-based storage platform.

Storage requirements for 3D AIGC

Our research focuses on perception and generation. In the 3D domain, task complexity is different from image and text processing. This placed higher demands on our AI models, algorithms, and infrastructure.

We illustrate the complexity of 3D data processing through a typical pipeline. On the left side of the diagram below is a 3D model containing texture (top-left) and geometry (bottom-right) information. First, we generate rendered images. Each model has text labels describing its content, geometric features, and texture features, which are tightly coupled with the model. In addition, we process geometry data, such as sampled points and necessary numerical values obtained from data preprocessing, like signed distance fields (SDFs). It's important to note that 3D model file formats are highly diverse, and image formats are also different.

image: 3D data processing pipeline
caption: 3D data processing pipeline

Our work spans language models, image/video models, and 3D models. As data volume grows, so does the storage burden. The main characteristics of data usage in these scenarios are as follows:

Language models: Data typically consists of a vast number of small files. Although individual text files are small, the total file count can reach millions or even tens of millions as data volume increases. This makes the management of such a large number of files a primary storage challenge.
Image and video data: High-resolution images and long videos are usually large. A single image can range from hundreds of kilobytes to several megabytes, while video files can reach gigabytes. During preprocessing—such as data augmentation, resolution adjustment, and frame extraction—data volume increases significantly. Especially in video processing, where each video is typically decomposed into a large number of image frames, managing these massive file collections adds considerable complexity.
3D models: Individual models, especially complex ones like point clouds, can be several gigabytes in size. 3D data preprocessing is more complex than other data types, involving steps like texture mapping and geometry reconstruction, which consume great computational resources and can increase data volume. Furthermore, 3D models often consist of multiple files, leading to a large total file count. As data grows, managing these files becomes increasingly difficult.

Based on the storage characteristics discussed above, when we chose a storage platform solution, we expected it to meet the following requirements:

Diverse data formats and cross-node sharing: Different models use different data formats, especially the complexity and cross-platform compatibility issues of 3D models. The storage system must support multiple formats and effectively manage data sharing across nodes and platforms.
Handling data models of different sizes: Whether it's small files for language models, large-scale image/video data, or large files for 3D models, the storage system must be highly scalable to meet rapidly growing storage demands and handle the storage and access of large-size data efficiently.
Challenges of cross-cloud and cluster storage: As data volume increases, especially with petabyte-level storage needs for 3D models, cross-cloud and cluster storage issues become more prominent. The storage system must support seamless cross-region, cross-cloud data access and efficient cluster management.
Easy scaling: The need for scaling is constant, whether for language, image/video, or 3D models, and is particularly high for 3D model storage and processing.
Simple operations and maintenance: The storage system should provide easy-to-use management interfaces and tools. Especially for 3D model management, operational requirements are higher, making automated management and fault tolerance essential.

Storage solutions: from NFS, GlusterFS, CephFS to JuiceFS

Initial solution: NFS mount

Initially, we tried the simplest solution—using NFS for mounting. However, in practice, we found that the training cluster and rendering cluster required independent clusters for mount operations. Maintaining this setup was very cumbersome. Especially when adding new data, as we needed to write mount points separately for each new dataset. When the data volume reached about 1 million objects, we could no longer sustain this approach and abandoned it.

image: Storage architecture based on NFS- difficult scaling, complex operations
caption: Storage architecture based on NFS- difficult scaling, complex operations

Mid-term solution: GlusterFS

GlusterFS was an easy-to-start-with choice, offering simple installation and configuration, acceptable performance, and no need for multiple mount points—just add new nodes.

While GlusterFS greatly reduced our workload in the early stages, we also discovered issues with its ecosystem:

Many GlusterFS execution scripts and features required writing custom scheduled tasks. Particularly when adding new storage, it had additional requirements, such as needing to increase nodes by specific multiples.
Support for operations like cloning and data synchronization was weak. This led us to frequently consult documentation.
Many operations were unstable. For example, when using tools like fio for speed testing, results were not always reliable.
A more serious problem was that GlusterFS performance would drastically decline when the number of small files reached a certain scale. For example, one model might generate 100 images. With 10 million models, that would produce 1 billion images. GlusterFS struggled severely with addressing in later stages, especially with an excessive number of small files. This led to significant performance drops and even system crashes.

image: Storage architecture based on GlusterFS
caption: Storage architecture based on GlusterFS

Final selection: CephFS vs. JuiceFS

As storage demands grew, we decided to use a more sustainable solution. After evaluating various options, we compared CephFS and JuiceFS.

Although Ceph is widely used, through our own practice and reviewing documentation, we found Ceph's operational and management costs to be very high. Especially for a small team like ours, handling such complex operational tasks proved particularly difficult.

JuiceFS had two native features that strongly aligned with our needs:

The client data cache. For our model training clusters, which are typically equipped with high-performance NVMe storage, fully utilizing client caching could significantly accelerate model training and reduce pressure on the JuiceFS storage backend.
JuiceFS' S3 compatibility was crucial for us. As we had developed some visualization platforms based on storage for data annotation, organization, and statistics, S3 compatibility allowed us to rapidly develop web interfaces supporting visualization, data statistics, and other features.

The table below compares basic features of CephFS and JuiceFS:

Comparison basis	CephFS	JuiceFS
File chunking	✓	✓
Metadata transactions	✓	✓
Strong consistency	✓	✓
Kubernetes CSI Driver	✓	✓
Hadoop-compatible	✓	✓
Data compression	✓	✓
Data encryption	✓	✓
Snapshot	✓	✓
Client data caching	✕	✓
Hadoop data locality	✕	✓
S3-compatible	✕	✓
Quota	Directory level quota	Directory level quota
Languages	C++	Go
License	LGPLv2.1 & LGPLv3	Apache License 2.0

Storage platform practice based on JuiceFS

Metadata engine selection and topology

JuiceFS employs a metadata-data separation architecture with several metadata engine options. We first quickly validated the Redis storage solution, which is well-documented by the JuiceFS team. Redis' advantage lies in its lightweight nature; configuration typically takes only a day or half a day, and data migration is smooth. However, when the number of small files exceeded 100 million, Redis' speed and performance significantly declined.

image: JuiceFS 架构图（第四版）-第 2 页-winfsp (2)
caption: JuiceFS Community Edition architecture

As mentioned earlier, each model might render 100 images. With other miscellaneous files, the number of small files increased dramatically. While we could mitigate the issue by packing small files, performing modifications or visualization on packed data greatly increased complexity. Therefore, we preferred to retain the original small image files for subsequent processing

As the file count grew and soon exceeded Redis' capacity, we decided to migrate the storage system to a combination of TiKV and Kubernetes (K8s). The TiKV-K8s setup provided us with a more highly available metadata storage solution. Furthermore, through benchmarking, we found that although TiKV's performance was slightly lower, the gap was not significant, and its support for small files was better than Redis'. We also consulted JuiceFS engineers and learned that Redis has poor scalability in cluster mode. Therefore, we switched to TiKV.

The table below shows read/write performance test results for different metadata engines:

	Redis-always	Redis-every second	MySQL	PostgreSQL	TiKV	etcd	FoundationDB
Write big files	730.84 MiB/s	731.93 MiB/s	729.00 MiB/s	744.47 MiB/s	730.01 MiB/s	746.07 MiB/s	744.70 MiB/s
Read big files	923.98 MiB/s	892.99 MiB/s	905.93 MiB/s	895.88 MiB/s	918.19 MiB/s	939.63 MiB/s	948.81 MiB/s
Write small files	95.20 files/s	109.10 files/s	82.30 files/s	86.40 files/s	101.20 files/s	95.80 files/s	94.60 files/s
Read small files	1242.80 files/s	937.30 files/s	752.40 files/s	1857.90 files/s	681.50 files/s	1229.10 files/s	1301.40 files/s
Stat files	12313.80 files/s	11989.50 files/s	3583.10 files/s	7845.80 files/s	4211.20 files/s	2836.60 files/s	3400.00 files/s
FUSE operations	0.41 ms/op	0.40 ms/op	0.46 ms/op	0.44 ms/op	0.41 ms/op	0.41 ms/op	0.44 ms/op
Update metadata	2.45 ms/op	1.76 ms/op	2.46 ms/op	1.78 ms/op	3.76 ms/op	3.40 ms/op	2.87 ms/op

Latest architecture: JuiceFS+TiKV+SeaweedFS

We use JuiceFS to manage the object storage layer. For the metadata storage system, we built it with TiKV and K8s. For object storage, we used SeaweedFS. This allows us to quickly scale storage capacity and provides fast access for both small and large files. In addition, our object storage is distributed across multiple platforms, including local storage and platforms like R2 and Amazon S3. Through JuiceFS, we were able to integrate these different storage systems and provide a unified interface.

image: Storage architecture- JuiceFS+TiKV+SeaweedFS
caption: Storage architecture- JuiceFS+TiKV+SeaweedFS

To better manage system resources, we built a resource monitoring platform on K8s. The current system consists of about 60 Linux nodes and several Windows nodes handling rendering and data processing tasks. We monitored read stability, and the results show that even with multiple heterogeneous servers performing simultaneous read operations, the overall system I/O performance remains stable, able to fully utilize the bandwidth resources.

Problems we encountered

During the optimization of the storage solution, we initially tried an erasure code (EC) storage scheme aimed at reducing storage requirements and improving efficiency. However, in large-scale data migration, EC storage computation was slow, and its performance was unsatisfactory in high-throughput and frequent data change scenarios. Especially when combined with SeaweedFS, bottlenecks existed. Based on these issues, we decided to abandon EC storage and switch to a replication-based storage scheme.

We set up independent servers and configured scheduled tasks for large-volume metadata backups. In TiKV, we implemented a redundant replica mechanism, adopting a multi-replica scheme to ensure data integrity. For object storage, we used dual-replica encoding to further enhance data reliability. Although replica storage effectively ensures data redundancy and high availability, storage costs remain high due to processing petabyte-scale data and massive incremental data. In the future, we may consider further optimizing the storage scheme to reduce costs.

In addition, we found that using all-flash servers with JuiceFS did not bring significant performance improvements. The bottleneck mainly appeared in network bandwidth and latency. Therefore, we plan to consider using InfiniBand to connect storage servers and training servers to maximize resource utilization efficiency.

Summary

When using GlusterFS, we could process at most 200,000 models per day. After switching to JuiceFS, the processing capacity increased significantly. Our daily data processing capacity has grown by 2.5 times. Small file throughput also improved notably. The system remained stable even when storage utilization reached 70%. Furthermore, scaling became very convenient, whereas the previous architecture involved troublesome scaling processes.

image: NFS vs. JuiceFS
caption: NFS vs. JuiceFS

Finally, let's summarize the advantages JuiceFS has demonstrated in 3D generation tasks:

Small file performance: Small file handling is a critical point, and JuiceFS provides an excellent solution.
Cross-platform features: Cross-platform support is very important. We found that some data can only be opened in Windows software, so we need to process the same data on both Windows and Linux systems and perform read/write operations on the same mount point. This requirement makes cross-platform features particularly crucial, and JuiceFS' design addresses this well.
Low operational cost: JuiceFS' operational cost is extremely low. After configuration, only simple testing and node management (for example, discarding certain nodes and monitoring robustness) are needed. We spent about half a year migrating data and have not encountered major issues so far.
Local cache mechanism: Previously, to use local cache, we needed to manually implement local caching logic in our code. JuiceFS provides a very convenient local caching mechanism, optimizing performance for training scenarios by setting mount parameters.
Low migration cost: Especially when migrating small files, we found using JuiceFS for metadata and object storage migration to be convenient, saving us a lot of time and effort. In contrast, migrating with other storage systems was very painful.

In summary, JuiceFS performs excellently in large-scale data processing, providing an efficient and stable storage solution. It not only simplifies storage management and scaling but also significantly improves system performance. This allows us to focus more on advancing core tasks. In addition, the JuiceFS tools are very convenient. For example, we used the sync tool for small file migration with extremely high efficiency. Without additional performance optimization, we successfully migrated 500 TB of data, including a massive number of small data and image files. It was done in less than 5 days, exceeding our expectations.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.

JuiceFS 2025 Recap: Sustaining Fast Growth, Scaling to Hundreds of Billions of Files

Wed, 31 Dec 2025 08:39:41 +0000

As we reflect on the journey of the past year, we're thrilled to share that 2025 ushered in the ninth year for JuiceFS Enterprise Edition and a significant fifth anniversary for our open-source Community Edition. Our focus remained unchanged: building a high-performance, easy-to-use file system.

All key metrics continued the growth momentum from the previous year. The data volume managed by the Community Edition grew by 89%, exceeding 1.3 EiB.

In 2025, the JuiceFS Community Edition continued to prioritize versatility, especially in supporting diverse AI workloads. We released Python SDK, improved Windows client usability, and strengthened integration with cloud-native ecosystems. Metadata engines like SQL databases and TiKV also received targeted optimizations. This year, alongside community contributors, we drove continuous iteration with 60 contributors, 305 new issues, and 601 merged pull requests (PRs).

During the development of the Enterprise Edition, our greatest challenge this year was managing hyperscale data. As AI technologies like autonomous driving become integrated into daily life, data volume growth is unprecedented. Managing hundreds of billions of files introduces exponentially increasing complexity in metadata management and data consistency. To tackle these challenges, the Enterprise Edition underwent comprehensive upgrades in core features like metadata partitioning and network performance.

JuiceFS Enterprise Edition 5.2, released in the first half of the year, already supports a single volume at the hundred-billion-file scale. The upcoming 5.3 version will push this limit to 500 billion files. This allows users to no longer worry about data scale, with JuiceFS' performance and stability providing solid assurance.

Let’s take a closer look at JuiceFS’ achievements in 2025.

Community Edition: Python SDK support and Windows client improvements

Since its open-source release, JuiceFS has been extensively validated in enterprise production environments, with its core features becoming increasingly stable. We released 9 versions throughout the year, with version 1.3 being the fourth major release since its 2021 open-source debut and designated as a long-term support (LTS) version. Key optimizations in this version include:

Python SDK Support, enhancing flexibility and performance in AI and data science scenarios.
Windows client optimizations, improving tool support and system service mounting capabilities.
Backup mechanism enhancement, enabling minute-level backups for 100 million files.
Integration with Apache Ranger, allowing JuiceFS to support fine-grained permission management in big data scenarios.
Performance improvements for SQL and TiKV metadata engines, delivering more efficient performance in hyperscale scenarios.

In the second half of the year, we began preparing for version 1.4. Planned new features include:

Support for user and group quotas
Redis client caching
Least recently used (LRU) cache support
SMB/CIFS support
Hadoop Kerberos support
S3 Gateway optimizations
Resumable sync tool transfers
Support for commercial data encryption algorithms
Readahead strategy optimization
Batch deletion improvements
Related tool optimizations

All aimed at further boosting system performance and stability.

image: 11
caption: JuiceFS contributors in 2025

JuiceFS CSI Driver released 18 versions over the past year, continuously optimizing JuiceFS' storage efficiency and stability in environments like Kubernetes. New features include volume path status detection, shared Mount Pods for the same file system, support for native Kubernetes Sidecar, and Dashboard cache group management. In addition, we made performance and reliability optimizations, not only improving stability but also enhancing compatibility with multi-pod configurations and containerized applications.

JuiceFS Operator added a scheduled cache warm-up feature to improve performance for application data access. It now supports cache groups deployed by replica, achieving cache high availability. It also introduced a Sync feature for efficient data synchronization within Kubernetes environments, ensuring consistency.

image: csi contributor
caption: JuiceFS CSI Driver contributors in 2025

Enterprise Edition: Single-volume hundred-billion-file scale with robust performance and stability

In the first half of 2025, JuiceFS Enterprise Edition 5.2 was released, breaking through the hundred-billion-file scale for a single file system and significantly enhancing stability for hyperscale clusters and the network performance of distributed caching. To achieve this, we spent a lot of time and effort, particularly in optimizing performance when handling massive datasets and high-concurrency access. This version has been validated in production environments across several enterprises, maintaining metadata latency at the 1-millisecond level even at the single-volume hundred-billion-file scale.
At the same time, we optimized distributed cache network performance, greatly reducing CPU overhead in TCP networks while improving network bandwidth utilization. In a test environment with 100 GCP 100Gbps nodes, aggregate read bandwidth reached 1.2 TB/s, close to full utilization of TCP/IP network bandwidth.
Furthermore, Python SDK achieved fsspec compatibility and on-demand import of object storage files, enabling easier access to existing data in object storage. This resolves read amplification issues in specific scenarios and enhances global QoS capabilities, thereby increasing system flexibility and performance.
The multi-zone architecture is a key technology enabling JuiceFS to handle hundreds of billions of files, ensuring high scalability and high-concurrency processing capabilities. In the second half of the year, we focused on developing version 5.3, which delivered comprehensive optimizations to this architecture. The zone limit was increased from 256 to 1,024, enabling a single volume to support the storage and access demands of over 500 billion files.
This achievement involved a series of complex tasks:

Systematically refining cross-zone link implementations and establishing a background self-check mechanism to improve cluster reliability and stability
Developing hotspot monitoring and automatic migration tools for efficient hotspot handling
Optimizing distributed cache management to reduce cache conflicts and improve concurrent performance

To further enhance distributed network performance, this version introduced RDMA technology for the first time (currently in experimental phase). Initial tests show it outperforms the TCP protocol in terms of stability and CPU usage. Version 5.3 is scheduled for release in January, 2026. Stay tuned for more details.

Community growth: Rapid expansion, total data volume exceeding 1.3 EiB

Currently, JuiceFS has got 12.6k+ GitHub stars. Our downloads have surpassed 50,000, while JuiceFS CSI Driver downloads have exceeded 5 million.The Slack community has reached about 1,000 members.
The fifth year since the open-source release of the Community Edition marks another year of rapid growth. User-reported data shows continued upward trends across key JuiceFS metrics:

File systems: 590k+, an 82% increase
Active clients: 150k+, a 46% increase
File count: 400+ billion, a 43% increase
Total data volume: 1.3+ EiB, an 89% increase

image: Number of file systems and data volume
caption: Number of file systems and data volume

This year, we made our mark at industry conferences by participating in events such as KubeCon + CloudNativeCon North America, Open Source Summit Japan, and SNIA Developer Conference (SDC).
To better support our users, we regularly host five sessions of Office Hours to introduce new features, address questions, and help users across various industries confidently deploy JuiceFS in production environments. Use cases span fields including autonomous driving, generative AI, AI infrastructure platforms, quantitative investing, and biopharmaceuticals. (View all case studies)
A special thanks to the users who shared their experiences this year—their practical insights have provided invaluable reference for the community:

JuiceFS+MinIO: Ariste AI Achieved 3x Faster I/O and Cut Storage Costs by 40%+, by Yutang Gao at Ariste AI
Zelos Tech Manages Hundreds of Millions of Files for Autonomous Driving with JuiceFS, by Junyu Deng at Zelos Tech
Why Gaoding Technology Chose JuiceFS for AI Storage in a Multi-Cloud Architecture, by Jia Ke at Gaoding Technology
StepFun Built an Efficient and Cost-Effective LLM Storage Platform with JuiceFS, by Changxin Miao at StepFun
INTSIG Built Unified Storage Based on JuiceFS to Support Petabyte-Scale AI Training, by Yifan Tang at INTSIG
vivo Migrated from GlusterFS to a Distributed File System Built on JuiceFS, by Xiangyang Yu at vivo
NFS to JuiceFS: Building a Scalable Storage Platform for LLM Training & Inference, by Wei Sun at a leading research institution in China
BioMap Cut AI Model Storage Costs by 90% Using JuiceFS, by Zedong Zheng at BioMap
JuiceFS at Trip.com: Managing 10 PB of Data for Stable and Cost-Effective LLM Storage, by Songlin Wu at Trip.com
How Lepton AI Cut Cloud Storage Costs by 98% for AI Workflows with JuiceFS, by Cong Ding at Lepton AI
Tongcheng Travel Chose JuiceFS over CephFS to Manage Hundreds of Millions of Files, by Chuanhai Wei at Tongcheng Travel

We've shared a remarkable year together. JuiceFS has evolved from an emerging open source project into a trusted solution powering AI-driven businesses today. We extend our deepest gratitude to every one of you for your active participation and steadfast support—whether through answering questions, sharing real-world experiences, or contributing code to the project.

In the coming year, JuiceFS will continue to deliver a more efficient and seamless experience for your work.

AI Data Storage: Challenges, Capabilities, and Comparative Analysis

Thu, 18 Dec 2025 08:59:33 +0000

Note: This article was first published on DZone and featured on its homepage.

The explosion in the popularity of ChatGPT has once again ignited a surge of excitement in the AI world. Over the past five years, AI has advanced rapidly and has found applications in a wide range of industries. As a storage company, we’ve had a front-row seat to this expansion, watching more and more AI startups and established players emerge across fields like autonomous driving, protein structure prediction, and quantitative investment.
AI scenarios have introduced new challenges to the field of data storage. Existing storage solutions are often inadequate to fully meet these demands.

In this article, we’ll deep dive into the storage challenges in AI scenarios, critical storage capabilities, and comparative analysis of Amazon S3, Alluxio, Amazon EFS, Azure, GCP Filestore, Lustre, Amazon FSx for Lustre, GPFS, BeeGFS, and JuiceFS Cloud Service. I hope this post will help you make informed choices in AI and data storage.

Storage challenges for AI

AI scenarios have brought new data patterns:

High throughput data access challenges: In AI scenarios, the growing use of GPUs by enterprises has outpaced the I/O capabilities of underlying storage systems. Enterprises require storage solutions that can provide high-throughput data access to fully leverage the computing power of GPUs. For instance, in smart manufacturing, where high-precision cameras capture images for defect detection models, the training dataset may consist of only 10,000 to 20,000 high-resolution images. Each image has several gigabytes in size, resulting in a total dataset size of 10 TB. If the storage system lacks the required throughput, it becomes a bottleneck during GPU training.
Managing storage for billions of files: AI scenarios need storage solutions that can handle and provide quick access to datasets with billions of files. For example, in autonomous driving, the training dataset consists of small images, each about several hundred kilobytes in size. A single training set comprises tens of millions of such images, each sized several hundred kilobytes. Each image is treated as an individual file. The total training data amounts to billions or even 10 billion files. This creates a major challenge in effectively managing large numbers of small files.
Scalable throughput for hot data: In areas like quantitative investing, financial market data is smaller compared to computer vision datasets. However, this data must be shared among many research teams, leading to hotspots where disk throughput is fully used but still cannot satisfy the application's needs. This shows that we need storage solutions that can handle a lot of hot data quickly.

The basic computing environment has also changed a lot.
These days, with cloud computing and Kubernetes getting so popular, more and more AI companies are setting up their data pipelines on Kubernetes-based platforms. Algorithm engineers request resources on the platform, write code in Notebook to debug algorithms, use workflow engines like Argo and Airflow to plan data processing workflows, use Fluid to manage datasets, and use BentoML to deploy models into apps. Cloud-native technologies have become a standard consideration when building storage platforms. As cloud computing matures, AI businesses are increasingly relying on large-scale distributed clusters. With a significant increase in the number of nodes in these clusters, storage systems face new challenges related to handling concurrent access from tens of thousands of Pods within Kubernetes clusters.

IT professionals managing the underlying infrastructure face significant changes brought about by the evolving business scenarios and computing environments. Existing hardware-software coupled storage solutions often suffer from several pain points, such as no elasticity, no distributed high availability, and constraints on cluster scalability. Distributed file systems like GlusterFS, CephFS, and those designed for HPC such as Lustre, BeeGFS, and GPFS, are typically designed for physical machines and bare-metal disks. While they can deploy large capacity clusters, they cannot provide elastic capacity and flexible throughput, especially when dealing with storage demands in the order of tens of billions of files.

Key capabilities for AI data storage

Considering these challenges, we’ll outline essential storage capabilities critical for AI scenarios, helping enterprises make informed decisions when selecting storage products.

POSIX compatibility and data consistency

In the AI/ML domain, POSIX is the most common API for data access. Previous-generation distributed file systems, except HDFS, are also POSIX-compatible, but products on the cloud in recent years have not been consistent in their POSIX support:

Compatibility: Users should not solely rely on the description "POSIX-compatible product" to assess compatibility. You can use pjdfstest and the Linux Test Project (LTP) framework for testing. We’ve done a POSIX compatibility test of the cloud file system for your reference.
Strong data consistency guarantee: This is fundamental to ensuring computational correctness. Storage systems have various consistency implementations, with object storage systems often adopting eventual consistency, while file systems typically adhere to strong consistency. Careful consideration is needed when selecting a storage system.
User mode or kernel mode: Early developers favored kernel mode due to its potential for optimized I/O operations. However, in recent years, we’ve witnessed a growing number of developers "escaping" from kernel mode for several reasons:
Kernel mode usage ties the file system client to specific kernel versions. GPU and high-performance network card drivers often require compatibility with specific kernel versions. This combination of factors places a significant burden on kernel version selection and maintenance.
Exceptions of kernel mode clients can potentially freeze the host operating system. This is highly unfavorable for Kubernetes platforms.
The user-mode FUSE library has undergone continuous iterations, resulting in significant performance improvements. It has been well-supported among JuiceFS customers for various business needs, such as autonomous driving perception model training and quantitative investment strategy training. This demonstrates that in AI scenarios, the user-mode FUSE library is no longer a performance bottleneck.

image: 221
caption

Linear scalability of throughput

Different file systems employ different principles for scaling throughput. Previous-generation distributed storage systems like GlusterFS, CephFS, the HPC-oriented Lustre, BeeGFS, and GPFS primarily use all-flash solutions to build their clusters. In these systems, peak throughput equals the total performance of the disks in the cluster. To increase cluster throughput, users must scale the cluster by adding more disks.

However, when users have imbalanced needs for capacity and throughput, traditional file systems require scaling the entire cluster, leading to capacity wastage.

For example, for a 500 TB capacity cluster using 8 TB hard drives with 2 replicas, 126 drives with a throughput of 150 MB/s each are needed. The theoretical maximum throughput of the cluster is 18 GB/s (126 ×150 = 18 GB/s). If the application demands 60 GB/s throughput, there are two options:

Switching to 2 TB HDDs (with 150 MB/s throughput) and requiring 504 drives
Switching to 8 TB SATA SSDs (with 500 MB/s throughput) while maintaining 126 drives

The first solution increases the number of drives by four times, necessitating a corresponding increase in the number of cluster nodes. The second solution, upgrading to SSDs from HDDs, also results in a significant cost increase.

As you can see, it’s difficult to balance capacity, performance, and cost. Capacity planning based on these three perspectives becomes a challenge, because we cannot predict the development, changes, and details of the real business.

Therefore, decoupling storage capacity from performance scaling would be a more effective approach for businesses to address these challenges. When we designed JuiceFS, we considered this requirement.

In addition, handling hot data is a common problem in AI scenarios. JuiceFS employs a cache grouping mechanism to automatically distribute hot data to different cache groups. This means that JuiceFS automatically creates multiple copies of hot data during computation to achieve higher disk throughput, and these cache spaces are automatically reclaimed after computation.

Managing massive amounts of files

Efficiently managing a large number of files, such as 10 billion files has three demands on the storage system:

Elastic scalability: The real scenario of JuiceFS users is to expand from tens of millions of files to hundreds of millions of files and then to billions of files. This process is not possible by adding a few machines. Storage clusters need to add nodes to achieve horizontal scaling, enabling them to support business growth effectively.
Data distribution during horizontal scaling: During system scaling, data distribution rules based on directory name prefixes may lead to uneven data distribution.
Scaling complexity: As the number of files increases, the ease of system scaling, stability, and the availability of tools for managing storage clusters become vital considerations. Some systems become more fragile as file numbers reach billions. Ease of management and high stability are crucial for business growth.

Concurrent load capacity and feature support in Kubernetes environments

When we look at the specifications of the storage system, some storage system specifications specify the maximum limit for concurrent access. Users need to conduct stress testing based on their business. When there are more clients, quality of service (QoS) management is required, including traffic control for each client and temporary read/write blocking policies.

We must also note the design and supported features of CSI in Kubernetes. For example, the deployment method of the mounting process, whether it supports ReadWriteMany, subPath mounting, quotas, and hot updates.

Cost analysis

Cost analysis is a multifaceted concept, encompassing hardware and software procurement, often overshadowed by operational and maintenance expenses. As AI businesses scale, data volume grows significantly. Storage systems must exhibit both capacity and throughput scalability, offering ease of adjustment.

In the past, the procurement and scaling of systems like Ceph, Lustre, and BeeGFS in data centers involved lengthy planning cycles. It took months for hardware to arrive, be configured, and become operational. Time costs, notably ignored, were often the most significant expenditures. Storage systems that enable elastic capacity and performance adjustments equate to faster time-to-market.

Another frequently underestimated cost is efficiency. In AI workflows, the data pipeline is extensive, involving multiple interactions with the storage system. Each step, from data collection, clear conversion, labeling, feature extraction, training, backtesting, to production deployment, is affected by the storage system's efficiency.

However, businesses typically utilize only a fraction (often less than 20%) of the entire dataset actively. This subset of hot data demands high performance, while warm or cold data may be infrequently accessed or not accessed at all. It’s difficult to satisfy both requirements in systems like Ceph, Lustre, and BeeGFS.

Consequently, many teams adopt multiple storage systems to cater to diverse needs. A common strategy is to employ an object storage system for archival purposes to achieve large capacity and low costs. However, object storage is not typically known for high performance, and it may handle data ingestion, preprocessing, and cleansing in the data pipeline. While this may not be the most efficient method for data preprocessing, it's often the pragmatic choice due to the sheer volume of data. Engineers then have to wait for a substantial period to transfer the data to the file storage system used for model training.

Therefore, in addition to hardware and software costs of storage systems, total cost considerations should account for time costs invested in cluster operations (including procurement and supply chain management) and time spent managing data across multiple storage systems.

Storage system comparison

Here's a comparative analysis of the storage products mentioned earlier for your reference:

Category	Product	POSIX compatibility	Elastic capacity	Maximum supported file count	Performance	Cost (USD)
	Amazon S3	Partially compatible through S3FS	Yes	Hundreds of billions	Medium to Low	About $0.02/GB/ month
	Alluxio	Partial compatibility	N/A	1 billion	Depends on cache capacity	N/A
Cloud file storage service	Amazon EFS	NFSv4.1 compatible	Yes	N/A	Depends on the data size. Throughput up to 3 GB/s, maximum 500 MB/s per client	$0.043~0.30/GB/month
	Azure	SMB & NFS for Premium	Yes	100 million	Performance scales with data capacity. See details	$0.16/GiB/month
	GCP Filestore	NFSv3 compatible	Maxmium 63.9 TB	Up to 67,108,864 files per 1 TiB capacity	Performance scales with data capacity. See details	$0.36/GiB/month
Lustre	Lustre	Compatible	No	N/A	Depends on cluster disk count and performance	N/A
	Amazon FSx for Lustre	Compatible	Manual scaling, 1,200 GiB increments	N/A	Multiple performance types of 50 MB~200 MB/s per 1 TB capacity	$0.073~0.6/GB/month
GPFS	GPFS	Compatible	No	10 billion	Depends on cluster disk count and performance	N/A
	BeeGFS	Compatible	No	Billions	Depends on cluster disk count and performance	N/A
	JuiceFS Cloud Service	Compatible	Elastic capacity, no maximum limit	10 billion	Depends on cache capacity	JuiceFS $0.02/GiB/month + AWS S3 $0.023/GiB/month

Conclusion

Over the last decade, cloud computing has rapidly evolved. Previous-generation storage systems designed for data centers couldn't harness the advantages brought by the cloud, notably elasticity. Object storage, a newcomer, offers unparalleled scalability, availability, and cost-efficiency. Still, it exhibits limitations in AI scenarios.

File storage, on the other hand, presents invaluable benefits for AI and other computational use cases. Leveraging the cloud and its infrastructure efficiently to design the next-generation file storage system is a new challenge, and this is precisely what JuiceFS has been doing over the past five years.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.