AI Data Storage: Challenges, Capabilities, and Comparative Analysis

Note: This article was originally published in 2025. It was revised in August 2026 to keep things up-to-date and is now also featured in the Packt DataPro newsletter.

Over the past five years, AI has advanced rapidly and has found applications in a wide range of industries. As a storage company, we've had a front-row seat to this expansion, watching more and more AI startups and established players emerge across fields like autonomous driving, computational biology, and quantitative investment. AI workloads have introduced new challenges to the field of data storage. Existing storage solutions are often inadequate to fully meet these demands.

In this article, we'll deep dive into the storage challenges posed by AI workloads, the critical capabilities required to address them, and a comparative analysis of the leading storage solutions spanning cloud and on-premises environments.

Storage challenges for AI

AI workloads have brought new data patterns:

• High-throughput data access challenges: In AI training pipelines, the growing use of GPUs by enterprises has outpaced the I/O capabilities of underlying storage systems. Enterprises require storage solutions that can provide high-throughput data access to fully leverage the computing power of GPUs. For instance, in smart manufacturing, where high-precision cameras capture images for defect detection models, the training dataset may consist of only 10,000 to 20,000 high-resolution images. However, each image is several gigabytes in size, resulting in a total dataset size in the scale of tens or hundreds of terabytes. If the storage system lacks the required throughput, it becomes a bottleneck during GPU-intensive AI training.

• Managing storage for billions of files: AI use cases need storage solutions that can handle and provide quick access to datasets containing billions of files. For example, in autonomous driving, the training dataset consists of small images, each about several hundred kilobytes in size. A single training set comprises tens of millions of such images, each sized several hundred kilobytes. Each image is treated as an individual file. The total training data amounts to billions or even 10 billion files. This creates a major challenge in effectively managing large numbers of small files.

• Scalable performance for hot data: In areas like quantitative investing, financial market data is much smaller in size compared to computer vision datasets. However, this data must be shared among many research teams, leading to hotspots where disk throughput is fully saturated but still cannot satisfy the application's needs. This indicates that we need storage solutions that can steadily serve a lot of hot data with high throughput and low latency.

The infrastructure landscape for AI has also changed dramatically.

These days, with cloud computing and Kubernetes getting so popular, more and more AI companies are setting up their data pipelines on Kubernetes-based platforms. Algorithm engineers request resources on the platform, write code in Jupyter Notebook to debug algorithms, use workflow engines like Argo and Airflow to plan data processing workflows, use Fluid to manage datasets, and use BentoML to deploy models as applications. Cloud-native technologies have become a standard consideration when building storage platforms. As cloud computing matures, AI applications are increasingly relying on large-scale distributed clusters. With a significant increase in the number of nodes in these clusters, storage systems face new challenges related to handling concurrent access from tens of thousands of pods within Kubernetes clusters.

The evolving application workloads and computing environments significantly change the landscape for professionals managing the underlying infrastructure and platforms. Existing hardware-software-coupled storage solutions often suffer from several pain points, such as no elasticity, no distributed high availability, and constraints on cluster scalability. Distributed file systems like GlusterFS, CephFS, and those designed for high-performance computing (HPC) such as Lustre, BeeGFS, and GPFS are typically designed for physical machines and bare-metal disks. While they can be deployed as large capacity clusters, they cannot provide elastic capacity and flexible throughput, especially when dealing with storage demands in the order of tens or hundreds of billions of files.

Key capabilities for AI data storage

Considering these challenges, we'll outline essential storage capabilities that are critical for AI scenarios, helping enterprises make informed decisions when selecting storage products.

POSIX compatibility and data consistency

In the AI/ML domain, POSIX is the most common API for data access. Previous-generation distributed file systems, except HDFS, are also POSIX-compatible, but products on the cloud have not been consistent in terms of their POSIX support:

• Compatibility: Users should not solely rely on the description "POSIX-compatible product" to assess compatibility. Instead, use pjdfstest and the Linux Test Project (LTP) framework for testing. We've conducted a POSIX compatibility test of cloud file systems for your reference.

• Strong data consistency guarantee: This is fundamental to ensuring computational correctness. Storage systems have various consistency implementations, with object storage systems often adopting eventual consistency, while file systems typically adhere to strong consistency. Careful evaluation is needed when selecting a storage system.

• User mode or kernel mode: Early developers favored kernel mode due to its potential for optimized I/O operations. However, in recent years, we've witnessed a growing number of developers "escaping" from kernel mode for several reasons:

  • Kernel mode usage ties the file system client to specific kernel versions. GPU and high-performance network card drivers often require compatibility with specific kernel versions. This combination of factors places a significant burden on kernel version selection and maintenance.

  • Runtime exceptions of kernel mode clients can potentially freeze the entire host machine, which is highly unfavorable for Kubernetes and other container-based platforms.

  • The user-mode FUSE library has undergone continuous iterations, resulting in significant performance improvements. It has been well-supported among JuiceFS customers for various business needs, such as autonomous driving perception model training and quantitative investment strategy training. This demonstrates that in AI scenarios, the user-mode FUSE library is no longer a performance bottleneck.

Linear scalability of throughput

Different file systems employ different principles for scaling throughput. Previous-generation distributed storage systems like GlusterFS, CephFS, the HPC-oriented Lustre, BeeGFS, and GPFS primarily use all-flash solutions to build their clusters. In these systems, peak throughput equals the total performance of the disks in the cluster. To increase cluster throughput, users must scale the cluster by adding more disks.

However, when users have imbalanced needs for capacity and throughput, traditional file systems require scaling the entire cluster, leading to wasted resources.

For example, a 500 TB primary-replica cluster using 8 TB hard drives would require 125 drives (500 TB × 2 ÷ 8 TB = 125 drives). Assuming each 8 TB drive delivers a throughput of 150 MB/s, the theoretical maximum throughput of the cluster would be 18.75 GB/s (125 × 150 MB/s = 18.75 GB/s). If the application demands 70+ GB/s throughput, there are two options:

• Switching to 2 TB HDDs (each still with 150 MB/s) and requiring 500 drives

• Switching to 8 TB SATA SSDs (each with 600 MB/s) while maintaining 125 drives

The first solution increases the number of drives by four times, necessitating a corresponding increase in the number of cluster nodes. The second solution, upgrading to SSDs from HDDs, also results in a significant cost increase. As you can see, it's difficult to balance capacity, performance, and cost. Resource planning based on these three perspectives becomes a three-way balancing challenge, because we cannot predict the development, changes, and details of the real application.

Therefore, decoupling storage capacity from performance scalability would be a more effective approach for organizations to address these challenges. When designing JuiceFS, we made this a core requirement.

In addition, handling hot data is a common problem in AI workloads. JuiceFS employs a cache grouping mechanism to automatically distribute hot data to different cache groups. This means that JuiceFS automatically creates multiple copies of hot data during computation to achieve higher disk throughput, and these cache spaces are automatically reclaimed after computation.

Managing massive amounts of files

Efficiently managing a very large number of files, for example, 10 billion files, requires the storage system to meet the following needs:

• Elastic scalability: For fast-growing organizations, the real-world situation is that storage needs often grow from tens of millions of files to hundreds of millions and then to billions. This kind of growth cannot be handled by just adding a few machines. Instead, storage clusters must support horizontal scaling by adding new nodes so they can keep up with business growth.

• Data distribution during horizontal scaling: When the system scales, using directory name prefixes to distribute data can sometimes cause uneven distribution across nodes. Thus, more sophisticated and yet efficient algorithms need to be implemented.

• Scaling complexity: As the file count increases, it becomes more important to consider how straightforward the system is to scale, how stable it remains, and what tools are available for managing the cluster. Some systems become much more fragile when they reach billions of files. It is essential that the storage system is not only scalable, but also easy to scale and highly stable.

Concurrent load capacity and feature support in Kubernetes environments

When reviewing storage system specs, some list a maximum limit for concurrent access. Users should run stress tests based on their actual workload needs. As the number of clients grows, quality of service (QoS) management becomes necessary. This includes controlling traffic per client and applying temporary read/write blocking policies when needed.

We must also note the design and supported features of the container storage interface (CSI) in Kubernetes. For example, the deployment method of the mounting process, whether it supports ReadWriteMany, subPath mounting, quotas, and hot updates.

Cost analysis

Cost analysis is a multifaceted concept, encompassing hardware and software procurement, often overshadowed by operational and maintenance expenses. As AI applications scale, data volume grows significantly. Storage systems must exhibit both capacity and throughput scalability, offering ease of adjustment.

In the past, the procurement and scaling of systems like Ceph, Lustre, and BeeGFS in data centers involved lengthy planning cycles. It took months for hardware to arrive, be configured, and become operational. Time costs, notably ignored, were often the most significant expenditures. Storage systems that enable elastic capacity and performance adjustments equate to faster time-to-market.

Another frequently underestimated cost is efficiency. In AI workflows, the data pipeline is extensive, involving multiple interactions with the storage system. Each step, from data collection, clear conversion, labeling, feature extraction, training, and backtesting to production deployment, is affected by the storage system's efficiency.

However, applications typically utilize only a fraction (often less than 20%) of the entire dataset actively. This subset of hot data demands high performance, while warm or cold data may be infrequently accessed or not accessed at all. It's difficult to satisfy both requirements in systems like Ceph, Lustre, and BeeGFS.

Consequently, many teams adopt multiple storage systems to cater to diverse needs. A common strategy is to employ an object storage system for archival purposes to achieve large capacity and low costs. However, object storage is not typically known for high performance, and it may handle data ingestion, preprocessing, and cleansing in the data pipeline. While this approach may not be the most efficient method for data preprocessing, it's often the pragmatic choice due to the sheer volume of data. Engineers then have to wait for a substantial period to transfer the data to the file storage system used for model training.

Therefore, in addition to hardware and software costs of storage systems, total cost considerations should also account for the time invested in cluster operations (including procurement and supply chain management) and the time spent managing data across multiple storage systems.

Storage system comparison

Here's a comparative analysis of the storage products mentioned earlier for your reference:

Conclusion

Over the last decade, cloud computing has rapidly evolved. Previous-generation storage systems designed for data centers couldn't harness the advantages brought by the cloud, notably elasticity. Object storage, a newcomer, offers unparalleled scalability, availability, and cost-efficiency. Still, it exhibits limitations when it comes to AI and other compute-intensive workloads.

File storage, on the other hand, presents invaluable benefits for such workloads. Leveraging the cloud and its infrastructure efficiently to design the next-generation file storage system is a new challenge, and this is precisely what JuiceFS has been doing over the past five years.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and the community on Discord.