JuiceFS Blog

How Zhuoyu Built a Petabyte-Scale Data Platform for Autonomous Driving with JuiceFS

Thu, 29 Oct 2026 05:36:23 +0000

Founded in 2016, Zhuoyu Technology (ZYT) focuses on the research, development, and large-scale application of mobile intelligence technologies. Leveraging our proprietary native multimodal foundation models and generative AI technologies, we build a unified mobile intelligence technology base that can efficiently adapt to multiple scenarios. It not only provides automakers with open, customizable, production-oriented intelligent driving assistance solutions (covering L2 and above), but also delivers underlying technology for autonomous mobile robots in diverse scenarios, empowering a full-spectrum intelligent mobility ecosystem from the ground to near-space.

As R&D scale expands, training and inference compute are gradually moving toward multi-data-center and hybrid-cloud environments, while data volume and access demands continue to grow. Compute resources can be scheduled on demand, but task startup can still be affected by mount status, origin data fetches, and model loading. How to make data ready in time and reduce compute waiting has become a key goal of storage architecture evolution.

We built a unified data foundation on JuiceFS. Through unified access, multiple storage pools, and shared caching, it supports data ingestion, capacity scaling, and hot data reuse. Combined with proactive warm-up and read optimization, it accelerates data delivery. Today, the online cluster carries 100-PB-scale data, with online read capability reaching the 100 GB/s range. This article covers the architecture evolution and production practices.

The autonomous driving data loop: growing volumes and repeated data consumption

Autonomous driving R&D revolves around a continuous data loop. Data collected by road-testing and production vehicles is filtered and labeled before entering model training. The resulting models then go through evaluation and deployment, while newly identified issues and newly collected data feed into the next iteration. At each stage, how quickly data becomes available directly affects whether compute resources have to wait and, ultimately, how efficiently R&D teams can iterate.

From a storage perspective, autonomous driving R&D is a data-intensive pipeline. Multimodal data continuously flows back into the platform and is repeatedly consumed by different R&D workloads. Data collected by production and road-testing vehicles, together with simulation-generated data, first enters shared multimodal storage. Downstream workloads—including data filtering, labeling, model training, simulation evaluation, and individual development environments—then consume the data.

image: Multimodal data flow R&D
caption: Multimodal data flow and consumption in autonomous driving R&D

As data volumes and iteration frequency increase, storage must support both continuous capacity growth and efficient data access to keep downstream workloads supplied with data.

Coupling between access, operations, and scaling in the original architecture

Before adopting JuiceFS, we used both S3 APIs and file-based access. Multiple access paths coexisted, storage configuration was scattered across applications, and compute-side teams were responsible for maintaining mount clients.

image: Multiple storage access paths and distributed configuration
caption: Multiple storage access paths and distributed configuration before the migration

We identified three core challenges:

Mounting coupled with workload scheduling: Task startup depended on mount clients being ready. Maintaining those clients on the compute side increased operational overhead and could leave compute resources waiting for storage.
Storage configuration coupled with applications: Storage details such as buckets, endpoints, and credentials were embedded in application code. Changes to underlying storage resources required application-side changes.
Capacity coupled with bandwidth scaling: Data volumes continued to grow while workloads repeatedly accessed hotspot data. Local caches could not be reused across nodes, making it difficult to scale storage capacity and read bandwidth independently.

Therefore, the new architecture needed to simplify application integration and mount management while allowing storage capacity and read bandwidth to scale independently.

Architecture upgrade: building a unified data platform on JuiceFS

Technology selection: S3 as the primary interface, JuiceFS as the storage foundation

To maintain compatibility with existing workloads while supporting future growth, we chose JuiceFS as the storage foundation for four main reasons:

Multiple access protocols: JuiceFS supports interfaces such as S3 and POSIX, allowing different workloads to use the access method that best fits their needs.
Capacity scaling: JuiceFS uses object storage as the underlying data store, providing room for continued data growth.
Cache acceleration: The Community Edition already provides local caching, giving us a foundation for further read-path optimizations.
Open source ecosystem: An active community makes it easier to continue improving the platform based on production requirements.

image: 1JuiceFS 架构图.drawio
caption: JuiceFS Community Edition architecture

Unified access layer: decoupling applications from storage

In the new architecture, applications access storage through an SDK provided by our internal data platform. The SDK provides a unified logical namespace and handles the mapping between logical paths and underlying storage locations and credentials. Applications integrated through the SDK primarily use S3 access, reducing the need to manage mount clients on compute nodes. Existing workloads can continue to use the POSIX interface.

image: Unified access layer and logical namespace based on the SDK
caption: Unified access layer and logical namespace based on the SDK

Using S3 as the primary access path also makes it easier to implement traffic isolation, monitoring, and auditing at the access layer.

The unified access layer separates application changes from storage changes, reducing the impact of underlying resource changes on application integration. We can continue scaling capacity, adding caches, and integrating external data sources while keeping the interfaces exposed to applications stable.

Building the core capabilities

We extended the JuiceFS Community Edition with multiple storage pools, distributed caching, and Under File System (UFS). The resulting architecture is shown below.

image: multiple storage pools, distributed caching, and UFS
caption: Architecture integrating multiple storage pools, distributed caching, and UFS

Multiple storage pools: scaling capacity and isolating failures

We added routing to the data layer of a JuiceFS volume. Based on policies such as file size and configured weights, the router distributes writes across different storage pools. The backends can be multiple S3 buckets or file storage systems.

image: multiple storage pools in a single volume
caption: Write routing and capacity scaling across multiple storage pools in a single volume

This gives us a one-to-many relationship between a JuiceFS volume and its underlying buckets. When an object storage resource reaches its scaling limit, we can add another storage pool to handle subsequent writes without migrating the existing data as a whole. Multiple independent storage pools also help reduce the impact of failures within any single resource domain.

Distributed caching: reusing hot data across nodes

Initially, we used local caching, which prevented nodes from sharing cached data. When a workload was scheduled to a new node and the required data was not available locally, the node had to fetch the data from the source again and keep another copy in its local cache. This increased read latency and reduced cache utilization.

To address this, we developed distributed caching of the Community Edition. Multiple clients can reuse hot data in a shared cache, while object storage remains the authoritative source of data. The read path works as follows:

The client uses consistent hashing to select a cache node.
If the data is available on the cache node's local disk, the node returns it directly.
If the data is not cached and fetching from the source is allowed, the cache node reads it from S3, returns the data to the client, and keeps a copy locally.

Local client-side caching remains available, giving us a multi-level caching architecture together with distributed caching.

image: Distributed cache read path
caption: Distributed cache read path

UFS: integrating external data and accelerating access

We also developed UFS of the Community Edition to bring external data sources into the JuiceFS namespace. UFS provides unified access to external data and accelerates reads through caching, while the authoritative copy remains in the original system.

For example, suppose an existing dataset is stored in S3 and we want to accelerate access using the JuiceFS cache. We can map the dataset to a path in JuiceFS without migrating the original data. Then, applications can access the data through the mapped path.

image: unified namespace with UFS
caption: Mapping external data sources into a unified namespace with UFS

The current implementation uses read-only mappings. It supports the same backend storage types as the Community Edition. Metadata can be loaded on demand or warmed up in advance. Even without warm-up, JuiceFS can retrieve directory and file metadata from the external storage as needed when a directory is accessed.

Production at hundreds-of-petabytes scale: architecture and optimizations

Our production clusters now manage hundreds of petabytes of data, deliver hundreds of GB/s of bandwidth, and provide petabyte-scale cache capacity. To improve data preparation and workload read performance, we optimized cache orchestration, model delivery, network transport, and data formats.

Cache orchestration: building caches as data arrives and proactively warming hot data

Based on statistics collected through our SDK, we found that roughly 80% of the data is accessed within eight days after it arrives. Therefore, we build distributed caches asynchronously as data is ingested and plan cache capacity and TTLs based on access patterns, so that workloads can read data from the cache. Cached data is reclaimed when it expires or when cache capacity becomes insufficient.

image: Cache lifecycle driven by hotspot windows and TTLs
caption: Cache lifecycle driven by hotspot windows and TTLs

For workloads that need explicit control over data preparation, we introduced Fluid for proactive orchestration. A Dataset describes the dataset and its source, while applications access the data through the corresponding Kubernetes PVC.

As part of the workload workflow, DataLoad works with the JuiceFS Runtime to warm the data. The application starts consuming the data only after it’s ready. The Fluid Controller coordinates resources and state throughout the process, managing the lifecycle from cache creation and application consumption to cache reclamation. For workloads that are not integrated with Fluid, we also provide APIs that allow applications to control cache warming and eviction themselves.

image: Fluid-based cache warm-up and lifecycle orchestration
caption: Fluid-based cache warm-up and lifecycle orchestration

Inference workloads: large-scale model delivery

For inference workloads, data preparation primarily means delivering models. An inference Pod cannot start serving requests until its model has been loaded. Previously, Pods loaded models directly from the model repository at startup. As the number of nodes increased, the repository had to handle a large number of repeated reads. Therefore, we placed a cache layer between the model repository and inference nodes.

With this cache layer, models are first warmed up into the distributed cache and then distributed over the cache network to the local caches on inference nodes. Requests for the same data block are routed to the same cache node through consistent hashing, allowing the cache layer to aggregate I/O and reduce repeated reads from the source.

When multiple inference Pods load the same model, the shared cache can aggregate their read requests and keep the amount of data fetched from the model repository close to the size of a single copy of the model. In our internal inference workloads, this approach improved deployment speed by more than 10× compared with the previous architecture.

image: Inference model delivery through shared caching
caption: Inference model delivery through shared caching

Training workloads: TCP zero-copy optimization

Once data is in the cache, workloads still need to continuously read it over the network. To reduce the overhead of this path, we optimized data transfer between clients and cache nodes, starting with TCP because it applies to a broad range of environments.

In the conventional read path, ReadAt copies data from kernel space into a user-space buffer, and then conn.Write copies it back into the kernel for transmission. When the requested data is available in cacheFile on the server and the requested range is valid, we can use io.CopyN → sendfile(2) to send the file data directly to the TCP socket. This eliminates this user-space copy.

image: Zero-copy TCP send path for cache hits
caption: Zero-copy TCP send path for cache hits

To evaluate the performance of the optimized cache path, we ran a 1 MiB sequential-read benchmark. The combined bandwidth across multiple servers was 1,200 Gbit/s. As client load increased, measured throughput reached approximately 135–138 GiB/s. At around 135 GiB/s, P99 latency for 1 MiB requests was approximately 2 ms. When throughput increased further to around 138 GiB/s, P99 latency rose to approximately 20–30 ms. This shows that as throughput approaches the bandwidth ceiling, a trade-off remains between throughput and tail latency.

image: Sequential read benchmark
caption: Sequential read benchmark

For the throughput-oriented MLPerf Storage UNet3D workload, we first used a high-density single-client topology. We concentrated the concurrent read load from eight accelerator ranks on a single node with a single 200 Gbit/s data ingress, allowing us to measure the data delivery ceiling of a single client. The tests covered three accelerator configurations: H20, H100, and B200. The H100 and B200 tests reached the limits of the 200G network and the client read path, shifting the bottleneck from backend storage to the client data ingress. In the next phase, we’ll extend the UNet3D training workload to multiple clients and combine it with RDMA to further evaluate sustained data delivery and accelerator utilization for large-scale, multi-node, multi-accelerator training.

image: Theoretical bandwidth requirements and measured results for MLPerf UNet3D
caption: Theoretical bandwidth requirements and measured results for MLPerf UNet3D

Lance columnar storage: reducing I/O at the data format layer

In addition to improving storage-path throughput, we also reduced the amount of data and number of I/O operations required by workloads by changing the data format.

For example, consider labeling data. A single asset may correspond to multiple JSON files, which were previously packaged into tar archives. Downstream processing still had to parse the JSON frame by frame, resulting in repeated serialization and format conversion and generating a large number of small I/O operations.

We switched the labeling data to Lance and flattened commonly used JSON fields into columns. Data is passed in memory through Arrow batches and written to Lance tables in batches. Downstream workloads select only the required columns and read them in batches, reducing unnecessary reads, per-frame JSON processing, and repeated serialization.

image: Per-frame JSON processing vs. batched data flow with Lance
caption: Per-frame JSON processing vs. batched data flow with Lance

The benefits vary across pipelines. I/O time decreased by approximately 20%–90%, IOPS requirements decreased by approximately 90%, and bandwidth consumption decreased by more than 60%. These improvements significantly reduced the demand that application workloads placed on storage resources.

Combining Lance with JuiceFS for targeted data warm-up

Column-based reads also allow us to make data warm-up more targeted. We use layout metadata such as the Lance Manifest to locate the physical byte ranges of the selected columns. Then, we map those ranges to the corresponding JuiceFS data blocks and warm only the required blocks, reducing unnecessary reads.

image: JuiceFS data blocks for targeted warm-up
caption: Mapping Lance columns to JuiceFS data blocks for targeted warm-up

These capabilities have been merged into JuiceFS Community Edition: byte-range data warm-up (PR #7398) and a tool for parsing Lance datasets (PR #7399). Together, they can generate a list of files and byte ranges based on a specified dataset version and selected columns, which can then be used for targeted data warm-up.

Conclusion

Looking back at this evolution, we built a unified data platform on JuiceFS and gradually improved it along three paths:

A stable access layer: The unified access layer abstracts differences between underlying storage systems. This minimizes the impact of storage scaling and deployment changes on application integration.
Shared hot data: Distributed caching allows data to be reused across nodes. This reduces repeated reads from the source when workloads move between nodes, clusters scale out, or multiple copies of the same data are accessed.
Proactive data preparation: Fluid orchestration and APIs allow data to be prepared before workloads consume it. This connects workload execution to the state of data preparation.

Together, these three approaches support data delivery at the hundreds-of-petabytes scale and performance requirements of hundreds of GB/s and above, while reducing the operational overhead and coordination required between storage and application teams. We continue to optimize cache orchestration, network transport, and data formats to make data available faster and reduce compute wait time, providing a more reliable storage foundation for autonomous driving R&D.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Scaling Data Management for Agent Sandboxes with JuiceFS

Thu, 22 Oct 2026 05:57:00 +0000

Recently, while working with cloud providers and agent teams to deploy sandboxes, we found that beyond getting data into the sandbox, we also need to consider how the data required by tasks and the data generated during execution can be continuously used across tasks and stages. Sandboxes can be quickly created and destroyed with tasks, but data often needs to be retained, shared, and transferred.

As sandbox concurrency scales further, the challenges go beyond data persistence and shared access. Operational issues such as client scale, cache reuse, small files, and fault isolation become more prominent.

This article combines our practice in agentic reinforcement learning (RL) scenarios and JuiceFS deployment experience in cloud provider sandboxes to introduce how sandbox data is organized, accessed across clouds, and mounted. We’ll also discuss the problems and optimization directions encountered under large-scale concurrency.

How to manage long-lived data in short-lived sandboxes

Take agentic RL as an example. A task typically goes through task distribution, sandbox execution, trajectory collection, reward evaluation, and training optimization. Data generated during task execution continues to be used by subsequent stages.

First, the compute lifecycle and data lifecycle separate. When an agent task starts, it needs to read datasets, code repositories, dependency environments, and model weights. During execution, it continuously produces trajectories, logs, intermediate files, and evaluation results. The sandbox can be destroyed after the task ends, but some of this data still needs to be retained for replay, analysis, evaluation, and subsequent training.

Second, data sharing and task isolation must coexist. Each sandbox typically needs an independent working directory and access boundary, but different agents, evaluators, or training pipelines may access the same dataset, code repository, or reuse results from previous tasks.

Third, data needs to flow across different compute environments. When tasks are distributed across self-built environments and sandboxes from different cloud providers, maintaining independent data copies in each environment not only increases synchronization and management costs but also limits the flexibility of task scheduling.

How JuiceFS manages and accesses sandbox data

To address these data management issues, we introduce how JuiceFS organizes and accesses data in actual sandbox environments.

Organizing shared data and task data with directories

Different customers adopt different data organization methods based on their application characteristics. Based on the usage patterns of existing customers, a typical approach is:

Datasets directory: stores core data such as evaluation datasets, training datasets, and benchmark task inputs.
Repo directory: stores code repositories, toolkits, and snapshots of related dependency environments.
Task directories: based on different tasks, runtime IDs, and training versions, independent directories are created in the file system to store data generated during each task run.

In this way, public data such as datasets and repos can be shared and accessed by multiple tasks, while each task has its own independent data space. At the same time, enterprises can decide which data needs to be retained long-term and which can be cleaned up after task completion, enabling more flexible data lifecycle management.

How cross-cloud sandboxes access the same data

Some customers use sandbox services from different providers across multiple cloud platforms simultaneously for reasons such as security, load balancing, and resource scheduling.

image: Multi-cloud compute access
caption: Multi-cloud compute access

Our customers' core data (such as training sets and inference data) is usually stored in their own data platform or file system, while sandbox environments from different cloud providers run on their respective cloud infrastructures. The challenge is how sandboxes on different clouds can access the same data without maintaining separate data copies.

In such scenarios, JuiceFS Enterprise Edition's cross-cloud data access capability can be used to keep unified data on the customer's own platform while allowing sandboxes in different cloud environments to access it on demand. With fine-grained permission control, data isolation can be achieved between sandboxes, and resources such as datasets, code repositories, and evaluation data on different clouds can be managed within a unified namespace.

For example, after data is written from cloud A, a sandbox in cloud B can quickly read it and continue the subsequent task flow. This improves the flexibility of task scheduling and reduces the complexity of data management in multi-cloud environments.

JuiceFS data mounting mode for strongly isolated sandboxes

To enable sandboxes to actually access this data, we also need to solve the problem of how to mount the file system into isolated environments.

Currently, sandbox solutions provided by cloud providers are different somewhat from self-built sandboxes in enterprises. From the existing cloud sandbox architectures, most are based on lightweight virtualization technologies such as Firecracker, with each cloud provider optimizing to different degrees.

The core characteristic of such sandboxes is that they create a strongly isolated environment inside a virtual machine. This keeps it isolated from the host and other sandbox instances. Therefore, under this strong isolation architecture, data access cannot simply rely on host shared directories.

To address this, the current approach is to run an independent JuiceFS client alongside each sandbox. The client is launched with the sandbox through a sidecar-like mechanism provided by the cloud provider and exposes a local mount path to the application sandbox.
In this mode, each sandbox connects to the backend JuiceFS service with an independent identity, accesses the metadata service deployed in the customer environment through the VPC network, reads data from the corresponding object storage, and mounts shared or exclusive directories as needed by the task. For data that needs to be shared across multiple tasks and components, a shared directory enables unified access, while data unique to each task is isolated and protected through independent subdirectories, ensuring data security.

For permission management, the token access control capability provided by JuiceFS Enterprise Edition can further enable fine-grained permission management, ensuring secure and controllable data access between different sandboxes.

JuiceFS in cloud provider sandboxes

We’ve implemented JuiceFS in sandbox environments on Tencent Cloud and Alibaba Cloud.

Tencent Cloud: data mounting methods in agent runtime

Tencent Cloud uses the agent runtime mechanism, with three levels of isolation: file system level, instance level, and storage level.

The first level is mounting block storage inside the sandbox. The block storage is exclusively used by a single sandbox, and the sandbox creates an independent file system on it. In this case, other environments cannot access this file system, achieving strong data isolation.
The second level is mounting storage services provided by Tencent Cloud itself. Through mount, the corresponding storage can be mounted into a specific sandbox environment, which is essentially mounting inside the sandbox.
The third level is the JuiceFS file system or subdirectory-level isolation. JuiceFS can mount the entire file system to a sandbox or only a specified subdirectory (subpath). Multiple sandboxes can share the same JuiceFS file system while dividing their data access scopes through different subdirectories, striking a balance between data sharing and task isolation.

image: Tencent Cloud agent sandbox file isolation solution
caption: Tencent Cloud agent sandbox file isolation solution

Alibaba Cloud: FC and ACS

Alibaba Cloud has two product lines:

The Function Compute (FC) Sandbox: Its implementation is similar to Tencent Cloud, also using a sidecar-like process mode to start a client in each sandbox to complete JuiceFS mounting.
The Container Compute Service (ACS): It also provides sandbox services but with a different implementation. ACS integrates JuiceFS mounting capability into its driver component. In actual use, customers decide whether and how to use this capability. The related code is also planned to be open-sourced in its components.

image: Alibaba Cloud FC agent sandbox
caption: Alibaba Cloud FC agent sandbox

Currently, whether on Tencent Cloud or Alibaba Cloud, JuiceFS mounting capability supports both Community Edition and Enterprise Edition.

Sandbox lifecycle management and mount control

In the current sidecar-like client mode, JuiceFS mounting is completed during sandbox startup, and the entire mount lifecycle is managed by the sandbox platform. For example, in Alibaba Cloud, the FC management platform is responsible; in Tencent Cloud, the agent runtime platform takes on this role.

Each sandbox corresponds to an independent JuiceFS client, which is created with the sandbox and completes directory mounting. The token, mount parameters, cache directory, and other configurations required during mounting are uniformly injected and managed by the runtime platform.

Under this mechanism, there are some differences between the Enterprise Edition and the Community Edition. JuiceFS Enterprise Edition provides more comprehensive management through the console and token capabilities, and cloud provider platforms have largely integrated these capabilities into their own mount management processes.

For the application side, the sandbox only needs to see the mounted data directory and does not need to directly manage the JuiceFS client. The creation, configuration, mounting, and exit of the client are all handled by the sandbox platform.

Challenges and optimization in large-scale agent sandbox scenarios

Client scaling bottleneck

Currently, one sandbox corresponds to one independent JuiceFS client, and each client is considered an independent client instance in the Enterprise Edition. As the number of sandboxes grows, the number of clients and the corresponding processes, connections, and metadata requests also increase.

However, most cloud provider sandbox implementations are based on Firecracker virtualization, and native Firecracker does not support passthrough mounting like VirtIO-FS. The number of clients a single file system can support is around 100,000. We’ve already received requests from some customers for millions or even more sandboxes. This makes client architecture optimization an important direction for future evolution.

One direction being explored is introducing a Mount Pod-like mode, where a JuiceFS client runs on the host or a designated node, and multiple sandboxes share the mounting capability. This can reduce the number of client instances and resource consumption, and further increase the supported sandbox scale.

Cache reuse and data loading efficiency

Currently, one sandbox corresponds to one independent JuiceFS client, and sandboxes may be distributed across different nodes. This makes it difficult to fully reuse client cache across tasks. Without a shared cache layer, a large amount of repeated data reads and loading pressure is directly transmitted to object storage; for cross-cloud sandbox scenarios, it may further increase bandwidth pressure on cross-cloud links.

To address this, we plan to combine task scheduling, host disk resources, and JuiceFS caching capabilities to improve cache reuse efficiency for hot data. For example, schedule related tasks to nearby nodes as much as possible, and use host disks to build a distributed cache, bringing frequently accessed data such as model datasets as close to compute as possible and reducing repeated loading from object storage.

When using cloud provider sandboxes, it’s necessary to combine the cache resources the cloud provider can offer; for self-built sandboxes, planning can be based on the customer's own infrastructure to balance performance, cache capacity, and resource cost.

Small files and metadata performance

During agent execution, data such as logs and trajectories often manifests as continuous small I/O writes, large numbers of small files, and frequent flushes. This places higher demands on JuiceFS metadata and write performance.

For such scenarios, we’ll work with application teams to explore optimization strategies, such as evaluating whether large numbers of small I/Os can be merged into larger-grained write requests, or using batch writes to reduce frequent small file operations. Adjusting the data write pattern on the application side can also help reduce pressure on the underlying metadata service.

Fault isolation and platform governance

In large-scale concurrency scenarios, it’s necessary to prevent a single abnormal task from affecting shared caches, metadata services, or object storage. For example, when abnormal access generates a large number of requests, its impact scope must be controlled as much as possible to avoid affecting other tasks.

If shared caches, shared clients, or client pools are further introduced, the original isolation boundaries will also change. After multiple sandboxes share underlying resources, fault isolation, resource limits, access permissions, and lifecycle management of shared clients need to be reconsidered.

On the other hand, as more platforms treat sandboxes as a public service for different teams or customers, data management will further extend to multi-tenant scenarios. Clear data and permission boundaries must be maintained between tenants, and different types of data require corresponding lifecycle policies.

Observability also needs to be improved accordingly, not only to observe client and cache status but also to understand overall storage usage, the running status of shared resources, and the scope of abnormal impact. This will provide a basis for subsequent resource governance.

From current practice, sandboxes are not difficult to create and destroy quickly; the real challenge is that the data behind them cannot disappear with them. Trajectories, logs, evaluation results, and training samples still need to be used by subsequent tasks, which makes storage gradually evolve from "providing a directory for the sandbox" to participating in the entire agent data pipeline over the long term.

As the scale grows from hundreds of thousands to millions, designs that were once simple and straightforward, such as sandbox clients and independent caches, also need to be reconsidered. We are still validating different approaches for client reuse, caching, and isolation, and we welcome exchanges with teams working on agent sandboxes, agentic RL, or related infrastructure about the problems they encounter.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

How to Access JuiceFS via S3 and WebDAV Protocols

Thu, 15 Oct 2026 01:11:00 +0000

Different applications access data in different ways: some rely on POSIX file interfaces, some are built on the S3 API, and some users want to browse and edit remote files through a file manager. JuiceFS' multi-protocol access capability is designed precisely to let these tools access the same file system.

In this article, we'll explain the shared initialization path of S3 Gateway and WebDAV, describe their applicable scenarios and common mappings, and dive into their implementation principles and boundaries.

A shared initialization path

JuiceFS consists of a metadata engine and object storage:

The metadata engine manages directories, attributes, locks, and sessions.
Object storage stores the actual file data blocks.

image: 1JuiceFS 架构图.drawio
caption: JuiceFS Community Edition architecture

From an implementation perspective, S3 Gateway and WebDAV first go through the same service initialization path. The service process creates a JuiceFS client based on the metadata address, listen address, and common parameters such as subdir, read-only, cache, limits, access-log, and metrics. Then, it initializes the metadata client, object storage, data block cache, and VFS configuration in sequence, ultimately obtaining an fs.FileSystem.

image: Shared service initialization path of S3 Gateway and WebDAV
caption: Shared service initialization path of S3 Gateway and WebDAV

In other words, the common foundation of the two is not a FUSE mount point, but a native JuiceFS client with sessions, caching, metrics, and background tasks. Before S3 requests and WebDAV requests enter their respective protocol adaptation layers, they already share the same JuiceFS file system and the same file system namespace.

Applicable scenarios and common mappings

The shared initialization path explains how the two connect to the same file system. In actual use, what matters more is choosing the access method: which scenarios are suitable for S3 Gateway, which are suitable for WebDAV, and how common requests map to JuiceFS operations.

Linux applications, training jobs, or workloads requiring full file system semantics should prefer FUSE/CSI.
Applications already built on the S3 SDK, AWS CLI, MinIO Client, or object storage toolchains should prefer S3 Gateway.
Desktop file manager access, DAV/rclone synchronization, Office/apps collaboration, and simple sharing should prefer WebDAV.

Mapping of common S3 Gateway APIs to JuiceFS file system operations:

S3 API	Purpose	Equivalent file system operation
GetObject	Download an object	open/read
PutObject	Upload or overwrite an object	create/write/rename
HeadObject	Query object metadata	stat/getxattr
DeleteObject(s)	Delete one or more objects	unlink/remove
ListObjects V1/V2	List objects by prefix	readdir+filter
CopyObject	Copy an object server-side	copy/read+write
MakeBucket	Create a bucket	mkdir
DeleteBucket	Delete an empty bucket	rmdir
NewMultipartUpload	Create a multipart upload session	create
PutObjectPart	Write a part	write
CompleteMultipart	Merge and commit the object	copy + atomic rename
Tags/Metadata	Read/write tags and metadata	get/setxattr

Mapping of common WebDAV methods to JuiceFS file system operations:

HTTP method	Purpose	Equivalent file system operation
GET	Download a file	read
PUT	Upload or overwrite a file	write/create
DELETE	Delete a file or directory	remove
MKCOL	Create a directory	mkdir
PROPFIND	Query properties or list a directory	stat/readdir
PROPPATCH	Modify custom properties	setxattr
COPY	Copy a file or directory	cp
MOVE	Move or rename	mv/rename
LOCK	Lock resources	File lock
UNLOCK	Unlock resources	Release a file lock
OPTIONS	Query server capabilities	Capability negotiation
HEAD	Query HTTP metadata only	Lightweight stat

S3 Gateway principles: from object semantics to file system operations

S3 Gateway converts S3 protocol requests into JuiceFS file system operations. This section covers four aspects:

The request path
Internal state
Key semantic conversions
Permission coordination

Request path: from the MinIO S3 server to `jfsObjects`

The MinIO S3 server handles the S3 protocol layer, including request routing, signature verification, authentication and authorization, and multipart upload orchestration. After requests pass through the protocol layer, they enter the ObjectLayer implemented by JuiceFS, namely jfsObjects. jfsObjects converts object-level reads, writes, listings, deletions, and attribute access into JuiceFS file system calls.

image: Call relationship among
caption: Call relationship among MinIO S3 server, jfsObjects, and JuiceFS file system

State storage: how the internal namespace carries the Gateway state

During request conversion, the Gateway does not only access user-visible object paths; it also uses an internal namespace to store state needed by the service itself. For example, .sys/tmp is used for temporary writes, .sys/uploads is used for multipart upload state, and .minio.sys is used to store MinIO/IAM-related metadata.

These internal paths are not application data in the ordinary user's understanding, but they are the foundation the Gateway needs to maintain upload processes, part states, and permission configurations.

Semantic conversion: `PutObject`, `Multipart`, `Bucket`, and `Metadata`

The S3 object model and the file system model are not completely the same, so the Gateway needs to perform semantic conversions at several key points.

PutObject must ensure that an object is visible only after the upload is complete. To prevent clients from reading incomplete objects, JuiceFS S3 Gateway writes to a temporary file and saves object attributes such as metadata and tag, commits to the target path via rename after completion. Leveraging the atomicity of file system rename, the Gateway can switch an object write from “uncommitted” to “committed.”

image: PutObject write flow
caption: PutObject write flow: from temporary files to atomic rename commit

Multipart upload needs to merge multiple parts into the final object. JuiceFS S3 Gateway can use file system capabilities such as CopyFileRange to combine already written parts and commit the final file via rename. This mainly adjusts file system metadata relationships rather than moving all data again.

image: Part merging and final commit flow
caption: Part merging and final commit flow of multipart upload

Bucket mapping handles the relationship between the object namespace and file paths. In single-bucket mode, the file system name corresponds to the bucket; in multi-bucket mode, top-level directories map to buckets. Object attributes such as metadata, tag, and ETag are stored through file extended attributes (xattr).

Permission coordination: IAM and multiple instances

S3 Gateway also needs to handle identity and permission information, including root users, regular users, policies, service accounts, and more. These configurations are stored in the JuiceFS file system through internal paths, allowing them to be reused across multiple Gateway instances.

Therefore, multiple Gateway instances can share the same set of IAM configurations; however, each instance caches IAM information, so permission changes are not synchronized instantaneously. In actual deployments, we need to consider the refresh cycle.

WebDAV principles: from file protocol to JuiceFS operations

WebDAV starts from a different point than S3 Gateway. It targets file access scenarios, with common clients including macOS Finder, DAV/rclone, and some Office/apps clients. Since the WebDAV protocol includes concepts such as files, directories, and attributes, it does not need to first handle the model conversion between buckets, objects, and keys on the one hand and paths and files on the other, unlike S3 Gateway.

Request path: from HTTP entry to `webdavFS`

After a WebDAV request enters JuiceFS:

1.It passes through net/http and indexHandler. This layer mainly handles HTTP entry-related capabilities, such as authentication, transport, and conversion of directory access requests.

2.The request enters x/net/webdav.Handler, which handles the WebDAV protocol layer and converts protocol methods into file access requests that webdavFS can process.

image: WebDAV request path
caption: WebDAV request path: from HTTP entry to JuiceFS file system operations

3.After entering webdavFS, the file access requests passed in by the Handler are converted into JuiceFS file system calls such as open, create, read/write, rename, delete, and attribute setting. Since the WebDAV protocol is close to file system semantics, the focus here is not to rebuild an object model, but to handle how request methods, file open flags, and attribute states map onto JuiceFS operations.

State boundaries: dead properties and LOCK

WebDAV adaptation involves not only data reads and writes but also the protocol's own state. Dead properties can be saved as JSON in xattr, thus landing in JuiceFS' file attribute system; however, LOCK state is different—it's currently kept in the service process memory. Single-instance deployments usually have no problem; in multi-instance deployments, different instances cannot automatically synchronize lock state.

These limitations usually do not affect Finder browsing, simple sharing, or DAV/rclone synchronization. However, if an application strongly relies on WebDAV file lock semantics, deployment mode and client behavior need to be verified separately.

S3 Gateway and WebDAV share the same JuiceFS client initialization path and access the same file system namespace. The difference is that S3 Gateway needs to convert object semantics into file system operations, while WebDAV is closer to file access semantics.

Therefore, the choice of access method should ultimately depend on the application itself: whether it organizes data as objects or operates on data as files and directories.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Nydus + JuiceFS: Reducing Container Startup Time for AI Inference from 116s to 1.4s

Fri, 09 Oct 2026 06:56:00 +0000

In large-scale AI inference services, when a cluster scales out, newly added inference instances must go through a series of cold-start steps before they can serve requests: container image preparation, file system mounting, runtime and inference framework initialization, and model weight loading. As image sizes and model weights continue to grow, the time spent on data preparation becomes increasingly prominent. During large-scale scaling events, many instances start simultaneously, further increasing the concurrent pressure on the registry, network, and backend storage.

To address these issues, the industry has developed several optimization strategies, including reducing image size, using P2P distribution to alleviate concurrent pull pressure, and employing on-demand loading to reduce data transfer on the critical startup path.
In this article, we’ll explore a shared‑storage‑based approach: combining Nydus’ on‑demand image loading capabilities with JuiceFS’ distributed storage and caching mechanisms. By sharing storage and reusing cached data, this approach reduces redundant data reads during large-scale scale-outs and shortens container startup time. For AI inference scenarios, container images and model weights can be managed together under a unified storage, caching, and prefetching system, accelerating both types of data while reducing the complexity of building and maintaining two separate data pipelines.

Why OCI image distribution is slow

Traditional Open Container Initiative (OCI) images are typically organized as compressed tar layers. In the image‑pull mode, a node must fully download and extract the required layers before the container can start. In practice, however, only about 6%–20% of files in an image are actually accessed during container startup; the rest of the download and extraction overhead is wasted on non‑critical‑path data.

In large clusters, this “prepare everything first, then start” approach causes three problems:

Long cold‑start times: For GB‑sized images (like AI inference images), full download and extraction significantly delay new instance startup, often taking minutes.
Registry bottleneck: During HPA‑triggered bursts or large‑scale rollouts, hundreds of nodes simultaneously request GB‑sized layers from the registry, easily overwhelming the registry and congesting the network.
Redundant network and storage usage: Without an efficient P2P or distributed sharing mechanism between nodes, the same layers are repeatedly transferred across the cluster, resulting in significant bandwidth consumption and duplicated storage.

Dragonfly+Nydus: what it solves and what it leaves behind

To mitigate registry bandwidth pressure during concurrent image pulls in large clusters, a common solution is to introduce P2P distribution systems like Dragonfly. The core idea is to build a P2P distribution network via a scheduler and node‑side agents, allowing nodes to share image data and reduce egress traffic from the registry.

However, P2P distribution primarily optimizes the data transfer path; it does not change the fundamental OCI mechanism of downloading and extracting layers before starting the container. Nydus, on the other hand, changes the loading approach: it transforms “prepare everything then start” into “fetch lightweight metadata first, and then read actual data on demand,” reducing data transfer on the critical startup path.

Combined, Nydus handles on‑demand loading, while Dragonfly distributes the on‑demand data, reducing both the actual image read volume and the concurrent access pressure on the registry.

However, this combination introduces additional infrastructure and operational complexity. Besides Nydus snapshotter, one must deploy and maintain Dragonfly’s scheduler, distributor, and node‑side components, and handle P2P network connectivity, cache space management, node anomalies, and cache eviction. In large‑scale GPU inference clusters, this means maintaining a separate data distribution and caching system independent of the storage layer.

This leads to an alternative thought: if the cluster already has a mature distributed file system like JuiceFS, could image data distribution and caching be offloaded to the shared storage system, reusing existing storage infrastructure while reducing the cost of building and maintaining extra P2P components?

Architecture redesign: Nydus blobs hosted by JuiceFS

The core idea of the Nydus+JuiceFS solution is to transform image data distribution—originally relying on P2P network scheduling—into shared storage and cache reuse based on a distributed file system. In this architecture, JuiceFS serves as the underlying unified data engine hosting Nydus image blob data, and worker nodes read data through the standard POSIX interface.

image: Nydus +JuiceFS
caption: Nydus +JuiceFS architecture and data flow

After an image is built and converted to Nydus format during the CI/CD phase, it’s split into two parts:

Bootstrap (metadata): Very small (typically a few hundred KB, for example, ~450KB), stored in the Image Registry, used for fast indexing.
Blobs (actual data blocks): Contains all file content (typically several GB, for example, ~1GB), persisted directly to the JuiceFS shared file system after conversion (for example, /var/lib/nydus/blobs).

When a worker node starts a container, containerd triggers nydus-snapshotter, and nydusd mounts the container’s read‑only layer via FUSE. Then, when the application issues actual read requests, nydusd looks up the index in the Bootstrap and reads the corresponding blob data on‑demand through the JuiceFS mount point.

The key change in this path is: the registry only distributes the manifest and the extremely lightweight bootstrap; the massive blob data transfer is entirely handled by JuiceFS’ distributed caching system. This fundamentally eliminates the registry network bottleneck during high‑concurrency pulls.

For AI inference scenarios, this architecture can further unify the storage paths for images and model weights. In addition to Nydus image blobs, model weights can also be stored in JuiceFS, allowing both types of data to reuse the same storage, caching, and prefetching mechanisms. This accelerates data loading while reducing the complexity of building and maintaining two data pipelines.

Comparison basis	OCI full image pull	Dragonfly+Nydus	JuiceFS+Nydus
Core concept	Full layer pull + layer extraction	P2P network offloading + on‑demand image loading	Storage‑layer decoupling + distributed caching + on‑demand loading
Cold start method & time	Full download and extraction required; typically 30s to 10min+	Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds	Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds
Image data transfer path	Registry → worker (full local download & extraction)	Registry / source → Dragonfly P2P network → worker on‑demand read	JuiceFS → worker on‑demand read via POSIX, accelerated by local & distributed caches
Core components	Registry + containerd	Nydus, Manager, Scheduler, Seed Peer, and dfdaemon	JuiceFS + nydus‑snapshotter
Image & model weight synergy	Image pulled from the registry to local node; model weights loaded via a separate path (NAS or object storage); two systems managed independently	Images loaded on‑demand via Nydus, distributed by Dragonfly; model weights still rely on independent external storage	Image blobs and model weights stored together in JuiceFS, sharing the same cache and warm-up mechanisms
Operational complexity	Easy to overwhelm registry bandwidth under high concurrency	Requires extra maintenance of P2P scheduler, node network, and caching system	Can reuse existing JuiceFS storage & cache infrastructure, reducing independent image distribution components
Local disk pressure	High: Stores full image and extracted data	High and unpredictable: Requires cache space for P2P; rapid scaling may cause disk full	Controllable: Unified shared storage pool, uses local NVMe for P2P distributed cache
Network requirements	Standard TCP	Requires widely open P2P ports across nodes; may be invalid in cross‑VPC/NAT environments	Standard LAN – only requires normal high‑bandwidth inter‑node LAN; no intrusive network requirements
Suitable scenarios	General workloads with no cold‑start requirements and small clusters (<20 nodes)	Ultra‑large clusters across data centers / clouds / edge computing environments without unified high‑bandwidth storage; strong ops capabilities of the team	Ultra-large clusters across data centers / clouds / edge computing environments without unified high-bandwidth storage, where JuiceFS is already deployed or can be deployed, aiming for unified image+model management and simplified ops

Runtime mechanism: how Nydus and JuiceFS coordinate

Startup phase: fetching bootstrap (metadata)

When the application starts, nydusd only needs to fetch the lightweight bootstrap file tree from the registry. The bootstrap describes the image file tree and block indexes, but contains no actual file content.

After the container root filesystem (rootfs) is mounted, the subsequent startup process proceeds. In our test scenarios, containers can be ready and pass health checks within hundreds of milliseconds, without transferring the full large image data before startup.

Read phase: on‑demand blob access

When the application process actually issues read operations (such as open() or read()), nydusd captures the I/O requests via the Linux FUSE mechanism and locates the corresponding blob data block and offset using the bootstrap index.

This process is completely transparent to upper‑layer applications. For the underlying data path, the read granularity changes from full image layers to the data blocks corresponding to specific files and offsets. nydusd directly accesses the JuiceFS mount point (for example, /var/lib/nydus/blobs) to read the target block, completely eliminating the runtime fetch‑from‑registry path.

Cache phase: data reuse via JuiceFS cache groups

JuiceFS’ distributed cache groups can reuse already‑cached blob data across the cluster. On a cache hit, data is returned to the kernel at near‑local‑NVMe speed. On a cache miss, only the specified offset block is pulled on‑demand and automatically filled into the cache. In addition to the automatic fill‑on‑miss, blobs can be pre-loaded into the cache to ensure full cache hits during on‑demand reads.

For AI inference scenarios, the same mechanism can also be applied to loading model weight files.

Production test data

In production, we compared the solutions above against a large AI inference image.

Container startup time comparison

The table below shows the test results for container startup time:

Scenario	OCI overlayfs	Nydus+JuiceFS	Comment
Cold pull + run	116 seconds	16 seconds	OCI uses public cloud registry; Nydus+JuiceFS uses JuiceFS + public cloud object storage
Cold pull + run (with distributed cache warm-up)	N/A	~1.4 seconds	Required blobs pre‑loaded in JuiceFS distributed cache
Warm pull + run	~0 seconds	0.22 seconds	OCI: same tag already on node; Nydus+JuiceFS: blobcache local cache

From the test results, we can see:

7x speedup without cache warm-up: When neither local nor distributed cache is warmed up, Nydus+JuiceFS reduces container startup time from 116s to 16s.
~99.9% reduction in data read volume during startup: In the cold pull + run scenario, read volume drops from 11 GB to 11 MB. Traditional OCI requires full download and extraction before startup, while Nydus+JuiceFS only fetches bootstrap metadata and reads blobs actually accessed during startup.
1.4s startup with distributed cache warm-up: When blobs are cached in JuiceFS distributed cache, startup takes 1.4s; with local blobcache warmed up (warm pull + run), startup takes only 0.22s.

On‑demand file read performance

The table below shows test results for: on-demand file read performance:

Scenario	Traditional OCI (local disk direct read)	Nydus+JuiceFS (distributed cache hit)
First‑block read latency (TTFB)	~0.9 milliseconds	~3.2 milliseconds
Small file / scattered configuration loading (IOPS)	Baseline	Close to baseline
Large file sequential read throughput	Local disk limit	GB/s level (JuiceFS NVMe distributed cache)

From the test results, we can see:

Time to first byte (TTFB): On distributed cache hit, first‑block read latency is ~3.2 milliseconds, about 2.3 milliseconds higher than local disk direct read (0.9 milliseconds). The extra latency comes from the FUSE data path and cache access, but remains at millisecond level, generally imperceptible to the application side.
Small file IOPS: Performance is very close to local disk baseline. In scenarios with dense Python dependency and configuration file reads, JuiceFS caching introduces no noticeable IOPS bottleneck.
Large file sequential throughput: On cache hit, Nydus+JuiceFS achieves GB/s sequential read throughput. Compared to traditional OCI, which is limited by single‑node local disk performance, JuiceFS can leverage NVMe distributed cache to provide high‑bandwidth data access.

Summary: combining on‑demand loading with shared cache

Nydus changes image loading from full pull to on‑demand reading, while JuiceFS places data blocks into a shared storage and distributed cache system. Together, they reduce container cold‑start time from 116s to 16s, and to ~1.4s with distributed cache warm-up; the data read volume during startup drops from 11 GB to 11 MB.

The value of this solution extends beyond just image startup acceleration. For AI inference scenarios, image blobs and model weights can be uniformly hosted by JuiceFS, sharing the same caching and prefetching mechanisms. During large‑scale scaling, distributed caching also reduces duplicate reads of the same data across many instances. This lessens concurrent pressure on the registry and backend storage.

In environments lacking shared storage, Dragonfly+Nydus is an excellent P2P alternative. However, in clusters that already have JuiceFS infrastructure, Nydus+JuiceFS delivers extreme image distribution and model loading acceleration with minimal operational overhead.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

JuiceFS Enterprise 5.4: From 100 Billion Files to 1 Million Clients

Tue, 29 Sep 2026 15:07:00 +0000

Following the support for 100-billion-file environments introduced in JuiceFS Enterprise Edition 5.3, v5.4 further improves JuiceFS across several dimensions for large-scale deployments. At this scale, even small per-operation overheads can add up to significant resource consumption. Once metadata is distributed across multiple zones, the system also needs to coordinate across nodes while maintaining performance, consistency, and stability.

To address these challenges, JuiceFS Enterprise Edition 5.4 introduces the following new capabilities and improvements:

Support for up to 1 million simultaneously mounted clients
Fast cloning of directories containing massive files
More than 90% RDMA bandwidth utilization
Up to 100% higher random-read performance under high concurrency
On-demand data synchronization for mirror file systems
More flexible caching strategies for a wider range of workloads
Multiple operational improvements for large-scale data

Supporting 1 million clients

As AI agents move toward large-scale deployment, the JuiceFS team faced a new challenge: supporting connections and concurrent access from up to 1 million clients on top of the metadata workload of a 100-billion-file file system. The team spent months exploring and optimizing the architecture, and introduced this capability in JuiceFS 5.4.

JuiceFS uses a distributed architecture that partitions metadata across multiple metadata nodes, allowing metadata for hundreds of billions of files to be managed across multiple zones. To allow these nodes to handle a large number of client connections at the same time, v5.4 co-locates metadata nodes with network proxies, enabling clients to connect to any metadata node.

The node that receives a request handles requests for its local partition directly. Requests involving other partitions are forwarded internally to the corresponding nodes. Multiple client sessions share the same physical connections between nodes, reducing the number of connections and the associated maintenance overhead.

Connection reuse is only part of the challenge. Querying large numbers of client sessions, reporting session status, and cleaning up expired sessions can also introduce significant overhead.

In v5.4, indexes are used to locate client sessions instead of scanning unrelated sessions. Paginated queries and sampled reporting reduce the overhead of session status management. Expired sessions are cleaned up in batches to prevent cleanup work from becoming concentrated in a single pass.

With these optimizations, JuiceFS 5.4 successfully handled 1 million simultaneous client connections in stress tests. Previously, when clients connected separately to multiple metadata nodes, the total number of connections could reach tens of millions. After optimization, 10 internal proxy nodes handled 1 million client connections, while connection reuse reduced the number of physical backend connections from the proxies to the metadata leader to about 400. This significantly reduced connection-management pressure on the leader.

image: 1-million-client test architecture
caption: 1-million-client test architecture

Fast cloning for large directories

When creating branches of training data or preparing test environments, users often need an independently modifiable copy of a directory. As directory sizes grow, cloning becomes more complex: in a multi-zone architecture, the metadata for a single directory tree may span multiple metadata zones, so cloning must coordinate subtree operations across nodes while preserving the complete directory hierarchy.

v5.4 introduces cross-zone directory cloning. It recursively processes subtrees while preserving the original cross-zone layout of the directory tree. Cloning only copies metadata. The underlying object data remains shared, so there is no need to copy it. Subsequent writes to the clone do not affect the source files.

In a deployment with 30 metadata zones, cloning 100 million files took 1 minute and 40 seconds, averaging 1 million files per second. More zones can provide greater parallelism, while actual performance also depends on metadata load and how the directory tree is distributed across zones.

Note that cross-zone directory cloning is not atomic. If new data continues to be written to the source directory during cloning, different subtrees in the cloned directory may reflect the source directory at different points in time. If strict consistency is required, pause writes to the source directory before starting the clone.

Performance improvements

More than 90% RDMA bandwidth utilization

Starting with JuiceFS 5.3, clients can use RDMA to communicate with distributed cache nodes, reducing CPU overhead during data transfers and improving transfer efficiency. v5.4 further improves bandwidth utilization and transfer stability.

In a test environment with a 400 Gbps RDMA NIC on the client and an 800 Gbps RDMA NIC on the distributed cache node, a single client achieved 45 GB/s of read throughput. A single distributed cache node delivered 90 GB/s of data throughput, with NIC utilization exceeding 90%.

Higher random-read IOPS under high concurrency

In multi-process training, HPC, and rendering scenarios, high-concurrency random reads can stress not only storage and network resources, but also client lock contention and request-processing overhead.

Over the past six months, the JuiceFS team has continuously optimized high-concurrency random-read performance by simplifying frequently executed code paths and reducing lock contention.

As a result, random-read IOPS has more than doubled overall compared with the previous implementation. With distributed caching, a single client reached up to 189,000 IOPS. With local caching, a single client reached up to 156,000 IOPS.

image: Random-read IOPS comparison
caption: Random-read IOPS comparison

Test environment: A dual-socket AMD server with 128 cores and 256 threads, 1 TB of DDR4 memory, a 100 Gbps NIC, and a 3.84 TB NVMe SSD used as the cache disk. The system ran Linux 5.14.

Optimizing data access across regions

On-demand synchronization to reduce data replication

In multi-region, multi-provider, and multi-cluster compute environments, a JuiceFS mirror file system can automatically synchronize files and directories from the source. Metadata is continuously synchronized to keep the directory structure and object versions up to date. Previously, object data was synchronized in full in the background by default.

However, a workload in the mirror region may only need a subset of the models, datasets, or historical directories stored at the source. For example, if the source contains 10 PB of data but the mirror accesses only 1% of it, full synchronization would transfer and store a large amount of data that the workload never uses.

v5.4 supports on-demand synchronization of object data for mirror file systems. When enabled, metadata continues to synchronize, while object data is no longer synchronized automatically in the background. When a client reads data, JuiceFS first checks the object storage in the mirror region. If the data has not been synchronized yet, JuiceFS fetches it from the source and stores it in the mirror on demand. This reduces unnecessary data transfer and storage consumption.

If an application requires local read performance on the first access, you can warm up the required data in advance. If a complete copy of the data is required, you can continue to use background synchronization.

Accelerating metadata access with read-only nodes

For workloads that only need to accelerate metadata access, v5.4 introduces a new mode that does not require creating a mirror file system.

Users can deploy read-only metadata service nodes in other regions to accelerate metadata reads for local clients. Clients continue to mount the source file system, while the acceleration nodes can scale dynamically to match access requirements.

More flexible cache management

Improving hot-data cache hit rates

Data from one-time scans is often unlikely to be accessed again. Caching this data can evict hot data that is accessed repeatedly. By default, when a request misses the distributed cache, the cache node fetches the data from object storage and stores it in the cache. v5.4 introduces a new cache-aside mode. When enabled, a cache miss causes the application node to read directly from object storage instead of filling the cache with the requested data.

Preserving cache hit rates when adding multiple cache nodes

When multiple cache nodes are added at once, read requests may be routed to new nodes that do not yet have the required data, triggering additional reads from object storage.

v5.4 introduces the --commission parameter. After a new node is enabled, it retains the node mapping from before scaling. If the data is not available in its local cache, the new node fetches it from the original node instead. This reduces reads from the source during cache scaling.

Reducing the number of files on large cache disks

On large-capacity cache disks, storing each cache block as a separate file can cause the number of files on the cache disk to grow continuously.

The new merge-cache mode combines multiple cache blocks into larger files and stores their indexes separately. This significantly reduces the number of files that the cache disk needs to manage.

This feature is currently available as a public beta.

Warming up specific byte ranges

If a workload reads only part of a large file, warming up the entire file consumes additional cache space and transfers unnecessary data.

v5.4 allows you to specify one or more byte ranges to warm up. JuiceFS only warms up cache blocks that intersect those ranges, aligning cache preparation with the data the workload actually needs to read.

This capability enables more precise warm-up for data formats such as Lance and Parquet.

Simplifying operations for massive files

Restoring files by deletion time

When restoring files from the trash, users may know approximately when an accidental deletion occurred but have difficulty identifying the target files by name or keyword alone.

The restore command now supports the start-time and end-time parameters. In addition to the existing keyword filters, these parameters allow users to restrict the restore operation by file deletion time.

Reducing GC and `fsck` memory usage

When checking large file systems, GC and fsck need to process large numbers of records, and sorting can consume significant amounts of memory.

v5.4 adds external sorting modes to both commands, using disk storage during sorting to reduce memory requirements. The external sorting mode for GC is available only with dry-run, allowing users to inspect and evaluate the operation without actually deleting data.

Restricting access with token allowlists

v5.4 supports access control through token allowlists. An allowlist can specify multiple first-level subdirectories under the root directory.

Clients using the token can only access and modify content within those subdirectories, allowing different teams or workloads to be restricted to their designated data.

Summary: scale changes everything

JuiceFS 5.3 brought the platform into a new phase of scale. At the same time, the rapid development of AI continues to push the boundaries of data volume and system load.

As file counts, client counts, and request volumes all reach new levels, performance, stability, consistency, and operational complexity are no longer independent concerns. They interact with one another and become increasingly difficult to manage as the system scales.

v5.4 addresses these system-level engineering challenges across large-scale deployments, covering client connectivity, cross-zone directories, data access, mirror synchronization, and cache management. JuiceFS Cloud Service users can now try JuiceFS Enterprise Edition 5.4 online. Users running on-premises deployments can contact the JuiceFS team for upgrade support.

JuiceFS is used across a range of demanding workloads, including LLMs and multimodal models, autonomous driving, embodied AI, and quantitative investment. We’ll continue working with organizations at the forefront of these fields to improve JuiceFS and build infrastructure that can keep pace with their applications.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Infinite Workspaces for AI Agents: A Practical Architecture with JuiceFS, SQLite, and Litestream

Wed, 23 Sep 2026 05:23:00 +0000

Most conversations about AI sandboxes start and end with boot speed and compute isolation: How fast does the container boot? Can the agent run untrusted code safely? But these discussions miss a fundamental truth: an autonomous agent turns a fresh machine into a rambling digital workspace. It clones code repos, installs heavy node_modules or Python virtual environments, downloads browser binaries, compiles artifacts, generates multi-gigabyte datasets, and writes massive trace logs.

"The files are not incidental. They are the working memory of the task."
– Aniket Maurya, Founder @ Celesto AI

The naive solution is to overprovision the root disk. However, determining the size of a boot disk before a task exposes its scope is an ineffective effort. A "quick bug fix" might require a full Next.js build chain, while a "data benchmark" might produce multi-gigabyte Parquet tables. Coupling system state to workspace state creates a bottleneck.

Instead, an emerging architectural pattern is radical in its simplicity: decouple the system disk from the durable workspace. Provide a POSIX-compliant mount that acts like an infinite, persistent drive, backed entirely by affordable object storage. But how do you build such a system without introducing complex services that defeat the purpose of ephemeral sandboxes? Recently, we co-hosted an Office Hours session with Celesto AI, a platform building secure sandboxes for autonomous AI agents, and the webinar was a delightful exploration of exactly this challenge.

Storage	Purpose
Root disk	Operating system, runtime, package manager internals, and system-level state
Workspace	Repositories, generated files, datasets, build artifacts, logs, and project state

Here is the creative engineering stack making this possible: JuiceFS as the file system layer, SQLite as the embedded metadata engine, and Litestream as the durability glue.

The creative trio: JuiceFS + SQLite + Litestream

Object stores (S3, GCS) are excellent for storing files and performing key-based lookups, but they are terrible at listing directories, handling stat calls, or managing millions of small files in batches (i.e., renaming a directory). You need a metadata engine to act as the "brain" of the file system, mapping paths to block locations.

Usually, JuiceFS deployments use centralized or clustered metadata services, such as Redis, TiKV, or FoundationDB. For isolated AI sandboxes, a creative alternative leverages the three familiar JuiceFS components working in harmony:

The JuiceFS client provides the POSIX bridge. It is responsible for the complex task of communicating with both the metadata engine and the object store. It is the "worker" that reads and writes the actual data, as well as performs administrative tasks such as status checks and data compaction.
SQLite acts as the metadata engine. Instead of connecting to a remote metadata server, JuiceFS also supports using a local SQLite database file as its metadata store. This means the entire file system metadata (every inode, directory entry, and block pointer) lives inside a single, standard .db file on the local disk.
Litestream provides the durability backing for that SQLite file. Litestream continuously takes SQLite snapshots and streams the SQLite write-ahead log (WAL) frames to the same object store.

Here is the magic: When the sandbox is running, JuiceFS uses the local SQLite file to resolve ls, cd, and read operations. The actual file blocks are already safe in the object store. (Although, the sync-interval configuration needs to be taken into consideration.) When the user stops the sandbox, the local SQLite file might be discarded, but Litestream has already replicated the metadata engine to the object store.

image: JuiceFS + SQLite + Litestream
caption: JuiceFS + SQLite + Litestream for AI agent workspaces

When the sandbox resumes on a fresh VM (or a different host entirely), the system simply downloads the latest SQLite snapshot from the object store and replays the WAL. Within seconds, the local SQLite can be fully reconstructed. JuiceFS mounts it, points back to the same object store blocks, and the agent is looking at the exact same workspace directory it left behind.

The JuiceFS file system doesn't care which computer it runs on. But it does remember its folder structure and file pointers with the portable SQLite snapshots and WAL frames. With this setup, we turned the working memory of the AI agent into files and blocks in the object store, making it durable and accessible instead of locking it to a specific boot disk or block device.

An unconventional but effective combo

JuiceFS supports interchangeable metadata engines. Choosing SQLite + Litestream subverts the typical playbook by prioritizing simplicity and per-agent isolation over global scale. Here is why this particular stack fits the AI agent use case so well:

Easy to set up: While it is not completely hands-off (you still manage the JuiceFS client, the local SQLite instance, and Litestream's replication configuration like Celesto AI does), this stack is drastically simpler to bootstrap than a distributed database cluster.
Per-agent isolation: Each sandbox gets its own dedicated SQLite metadata file. This eliminates noisy-neighbor problems. Metadata contention simply does not happen across agents, because each one has its own independent "brain" living right next to its mount.
Performance that fits the scope: A single agent workload typically produces tens of thousands to a few million files. SQLite handles this range comfortably with low latency and minimal resource footprint. While this approach obviously does not scale to the hundreds of billions of files that JuiceFS Enterprise Edition can manage globally, it falls squarely into the comfortable zone for individual agent workspaces.

Of course, this approach comes with important trade-offs and considerations:

No cross-agent sharing: The file system is inherently tied to a single SQLite instance. Agents cannot mount and share the same workspace concurrently. If your use case requires collaborative access or shared datasets across sandboxes, you would need a different strategy.
Resume latency depends on object store speed: When a sandbox resumes, the system must download the latest SQLite snapshot from the object store and replay the WAL frames. This adds noticeable startup time compared to a hot-mounted local disk. Tuning Litestream's sync-interval involves a direct trade-off between durability guarantees and resume performance.
Single-writer limitation: SQLite is fundamentally a single-writer database. Heavy parallel writes from multiple processes within the same sandbox can become a bottleneck. For autonomous agents following a largely sequential workflow, this limitation is rarely a practical issue. But it is worth keeping in mind for highly concurrent, multi-threaded agent designs.

The stack is not a universal solution. It is a deliberate, opinionated choice that trades global scalability and shared access for extreme simplicity and tight per-agent isolation. And for the AI sandbox use case, that trade-off often makes perfect sense.

The performance reality

This architecture is beautiful, but it isn't magic. You are trading local SSD latency for object storage durability and infinite scale. The benchmarks reveal a clear pattern:

Small files need attention: Writing 5,000 tiny 4 KiB files to an object-backed file system involves heavy metadata churn. Even with SQLite's fast local access, every close() and fsync() triggers WAL writes and Litestream replication. In these scenarios, the local root disk (ext4/XFS) will outperform the durable mount by a factor of 2x to 5x. See Celesto's blog post for more details.
Large sequential workloads shine: For datasets, model checkpoints, build artifacts, and logs (files > 10 MB), the object-backed mount achieves upwards of 200 MiB/s, which is nearly 90% of the performance of the local disk. The background block uploads are streamed efficiently. See Celesto's blog post for more details.

The golden rule for agents is to use the object-backed mount drive for durable, long-term state reports and large artifacts. Use the local ephemeral disk for high-churn, temporary scratch work that doesn't need to survive a restart. This hybrid approach gives you infinite capacity without sacrificing performance where it truly hurts.

Final thoughts

Cloudflare CFO Thomas Seifert recently predicted that within five years, non-human traffic could be as much as 1,000 times human traffic, making humans "a rounding error on the internet." He caveated that he's "called it wrong at every point along the way," but the direction is hard to dispute: AI and machine traffic already overtook human traffic in May 2026.

image: Cloudflare exec
caption: “Humans will be a rounding error on the internet” says Cloudflare exec

If that is true for network packets, it is likely true for storage as well.

Every autonomous agent session produces a workspace. Multiply that by millions of agents running continuously, resuming, retrying, and branching. Human developers generate storage in bursts tied to working hours. Agents generate it constantly, at machine speed.

This raises an uncomfortable question: if non-human consumers are about to dominate storage demand, are we designing for the right consumer? The combo of JuiceFS + SQLite + Litestream is one answer, but not the only one. Other approaches worth watching include object storage as the primary interface, shared metadata engine with concurrent access at agent scale (i.e., the JuiceFS Enterprise Edition), copy-on-write workspace snapshots, and many others.

Portable, shareable, scalable file storage for AI agents is still in its opening moves.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Keeping GPUs Fed: How Meta's AI Storage Architecture Aligns with JuiceFS

Thu, 17 Sep 2026 03:32:00 +0000

Introduction

Recently, I came across an eye-opening blog post and presentation from Meta's engineering team detailing their AI storage blueprint at scale. Meta operates hundreds of exabyte-scale storage clusters serving everything from Facebook and Instagram to Meta AI.

What caught my attention wasn't just the scale—it was the architecture. After years of wrestling with AI's brutal latency and throughput demands, Meta has converged on a clear pattern: data-metadata separation, rich client design, and comprehensive caching.

Here's the fascinating part: this blueprint isn't unique to Meta. JuiceFS shares these same first principles of design. The hyperscaler's new construction closely resembles the cloud-native architecture that JuiceFS has advocated for years.

In this post, I'll analyze both architectures, compare their core components, dive into their cross-region capabilities, and explain why this convergence in architecture makes the most sense for a distributed file system.

Why storage matters for AI workloads

Before diving into the architectures, it's worth understanding why storage has become such a critical component for AI infrastructure.

Meta puts it succinctly: "If AI is the brain, storage is the memory: capability and speed are highly dependent on the size of memory and speed of retrieval." Yet while AI compute performance has roughly tripled every two years, storage and interconnect performance growth have been far more modest. This disparity has made storage bottlenecks one of the primary contributors to GPU stalls for AI workloads, which is directly impacting both expenditures and time to market.

image: Dataloading across two GPUs
caption: Dataloading across two GPUs (Reference: Meta)

The math is brutal. During model training, hundreds of thousands of GPUs iterate over vast datasets across multiple epochs. GPUs synchronize their state periodically. If a single GPU stalls due to slow storage, it slows down the entire training job. In Meta's presentation, if GPUs were idle for about 20% of the time, "that 20% stall means about tens of millions of dollars loss every hour" for a data center the size of Manhattan.

Beyond the direct dollar cost, there is an even more consequential dimension: opportunity cost. The AI industry is moving at an unprecedented pace. Major model releases, which took roughly 104 weeks between 2020 and 2022, have now been compressed to approximately 4-week cycles by 2026. Every week of delay in shipping a frontier model translates to lost market share, diminished competitive advantage, and missed revenue opportunities.

In short, storage is what makes the difference between a model shipping in weeks versus months and between a billion-dollar data center running at 80% versus 100% efficiency.

Meta's AI storage blueprint: the evolved architecture

After iterating on their storage infrastructure, Meta has arrived at a definitive architectural blueprint to solve the GPU stall problem at its root while continuing to support existing external and internal products. The training stack has been gradually migrating to the BLOB-storage interface. The motivation is more about performance: AI workloads demand predictable, bounded pMax latency. Traditional layered metadata lookups could add hundreds of milliseconds per request, stalling GPUs during training. By moving to a BLOB-centric interface, Meta leverages flash drives to deliver the low, predictable latencies required to keep thousands of GPUs productive.

The three-component architecture

The diagram below illustrates the request flow for Meta's BLOB storage getObject API and shows the system's three major components.

image: BLOB storage getObject request flow
caption: BLOB storage getObject request flow (Reference: Meta)

The Tectonic data storage: At the foundation sits Tectonic, a regional, multi-tenant storage fabric providing high durability and availability through erasure coding. It supports tiering across HDDs and flash drives, intelligently managing hot, cold, and warm data placement.
The BLOB server and metadata layer: Operating on top of Tectonic, the BLOB layer handles API requests, stores metadata, and exposes a global, infinitely scalable storage fabric with policies allowing users to trade off between durability and availability, abstracting away the complexity of the underlying block storage.
The rich client SDK: Meta eliminated the data-plane proxy and built a rich client SDK with a Tectonic BlockClient embedded within it, enabling direct data reading from Tectonic storage servers to clients. The client also implements hedged reads and dynamic concurrency control to reduce tail latencies.

To maximize cache efficiency, Meta also leverages spare memory on GPU hosts as a distributed data cache by integrating peers from the Owl subsystem directly into the BLOB-storage client SDK, which yields an average cache hit rate of 80%.

Cross-region data access with tiered caching

Meta's cross-region approach treats storage as a global data lake, but with tiered caches accelerating access across geographical boundaries.

image: Cross-region data access with multi-tier caching
caption: Cross-region data access with multi-tier caching (Reference: Meta)

Cache Tier	Location	Role
L1 Cache	GPU host memory (RAM)	Hottest, most frequently accessed data
L2 Cache	GPU host flash (SSD)	Local flash tier for warm data
L3 Cache	Regional flash-based BLOB storage	Regional cache before falling back to global BLOB storage

The results speak for themselves: average ingestion time dropped by 93%, and worst-case ingestion time dropped by 97%. This is what it looks like when storage stops being the bottleneck and starts enabling AI research velocity.

JuiceFS: cloud-native, AI-native from day one

For readers already familiar with JuiceFS, the parallels to Meta's design are immediately apparent. The architecture Meta converged upon after years of iteration shares the same core principles that have guided JuiceFS from day one.

JuiceFS was architected from the ground up on the data-metadata separation principle, which also consists of three core components:

Object storage: JuiceFS uses object storage as the reliable, scalable backend for actual data storage. It supports all major public cloud offerings (AWS S3, Alibaba OSS, Google Cloud Storage, etc.) as well as self-hosted solutions like Ceph and MinIO. This layer ensures that AI datasets, which can be as large as petabytes or exabytes, are safe, accessible, and can grow almost without limit.
Metadata engine: The metadata engine stores all file system metadata, including file names, sizes, permissions, directory structures, and the mapping between files and their underlying data blocks. JuiceFS supports multiple engine options, including Redis, MySQL, PostgreSQL, TiKV, and many others. For extremely large-scale deployments, JuiceFS Enterprise Edition offers a proprietary Raft-based distributed metadata engine that provides high availability and strong consistency across multiple nodes.
JuiceFS client: The client handles all file I/O logic, including data slicing, merging, caching, and POSIX semantics enforcement. It communicates with both the metadata engine and object storage directly, coordinating reads and writes intelligently.

Cross-region strategy: mirrors and replication

The JuiceFS Enterprise Edition also provides a native approach to support cross-region and cross-cloud data access:

Mirror file systems: Deploy a full metadata cluster and optionally an object storage replica in a mirror region. Metadata syncs automatically from the source region with approximately 1-second latency under normal network conditions.
Async data replication: JuiceFS supports cross-region and cross-cloud asynchronous replication with a one-to-many topology. Data written in the primary region is replicated to multiple mirror regions in the background, enabling global datasets without manual copying.
Distributed cache: JuiceFS clients on mounting hosts can form a distributed cache group using a consistent hashing ring, sharing cached data blocks with one another. This is particularly effective for model training, where datasets are repeatedly accessed across GPU nodes.
Smart read strategy: Clients preferentially read from the local region. If the data isn't fully synced yet, they seamlessly fall back to the source region.

image: JuiceFS mirror file system architecture
caption: JuiceFS Enterprise Edition mirror file system

This design resonates with Meta's global data lake + tiered cache approach but offers a more flexible, cloud-native model that works across public clouds, private data centers, and hybrid environments.

Architectural similarities

On the surface, Meta mainly exposes a BLOB-style interface, while JuiceFS primarily provides a POSIX file system. Yet beneath these protocol differences, the architectural parallels are noteworthy. Both systems converged on the same three-component design with comprehensive caching and cross-region capabilities:

Dimension	Meta's BLOB-Storage	JuiceFS
Data storage layer	Tectonic: regional block storage fabric with erasure coding, tiering across HDDs and flash	Object storage: S3, OSS, GCS, Ceph, MinIO, and others
Metadata layer	Unified metadata scheme backed by ZippyDB with O(1) chunk lookups	Metadata engine: Redis, MySQL, PostgreSQL, TiKV, JuiceFS Enterprise Edition
Rich client	"Fat client SDK" with embedded Tectonic BlockClient, streaming directly from storage servers	JuiceFS client: handles all I/O logic, slicing, merging, caching; supports FUSE, POSIX, Hadoop SDK, CSI, S3 Gateway
Distributed cache	Tiered caching layers	Memory + local cache + distributed cache group
Cross-region	Tiered caching with global BLOB storage data lakes	Primary region + mirror file systems

A concrete example: explicit cache hydration

One interesting example is how both systems handle proactive cache hydration, which is the practice of loading data into the cache before it's actually needed.

Meta exposes a prefetch() API through the BLOB-storage SDK, while JuiceFS provides a warmup subcommand. Both serve the same purpose: allowing the client side to signal to the storage system exactly which data will be needed in the near future so it can be preloaded into cache before the GPU finishes its current work.

This is especially critical in distributed training, where thousands of GPUs may simultaneously read the same dataset. Without cache hydration, the initial requests would have to read from the underlying storage (Tectonic for BLOB and object store for JuiceFS) with higher latency. With prefetch() or juicefs warmup, data is staged in the distributed cache before the GPUs need it. In essence, both systems recognize the limitations of network-based storage and passive caching, and they both provide explicit prefetching as an important tool to compensate for what the storage layer alone cannot guarantee.

Conclusion

Meta's AI storage blueprint is a testament to the architectural patterns that emerge when a hyperscaler evolves its data infrastructure to hundreds of exabytes. After years of iteration, they arrived at a design that's similar to what JuiceFS has championed from day one: data-metadata separation, a rich client that handles I/O logic directly, tiered caching, and cross-region capabilities.

The same architectural pattern is also available to teams of any size. JuiceFS brought it into the big data era and now brings it into the AI age, with the flexibility of multi-cloud cross-region strategies, distributed caching, and a rich ecosystem of access protocols.

Hopefully, storage is no longer the bottleneck in your AI training stack. At least, it doesn't have to be.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Everything Is a File for Agents: Hello Uses JuiceFS to Build Stateful AI Workspaces

Thu, 10 Sep 2026 06:29:00 +0000

Hello is a Chinese mobility and technology company, offering car-pooling, car-hailing, battery swapping, shared bike, e-bike, and e-scooter rental services. We have 800 million+ registered users.

As our AI business continues to grow, we need to support multiple workloads simultaneously, including model training, data processing, and AI agents. To address issues such as data fragmentation, inconsistent access methods, and complex cross-cloud data flows caused by multiple storage systems, we introduced JuiceFS (an open-source, high-performance distributed file system) in production to build a unified file data foundation that connects Notebook, training, evaluation, and agent environments.

Among these workloads, agents have become an important application type and are already used in several real-world application scenarios. We’ve also deployed an internal personal agent for employees. Unlike training workloads, which primarily focus on high-performance I/O and access to massive numbers of files, agents place greater emphasis on data movement during task execution, state persistence, and the accumulation of results.

This article focuses on how we use JuiceFS to persist user profiles, conversations, memories, skills, execution traces, and application artifacts, and how a shared workspace decouples agent state from compute Pods.

Hello's AI infrastructure and storage landscape

Our business scenarios are diverse, including customer service, review, scheduling, analytics, and broadcasting. To support AI applications across these businesses, we’ve progressively built an AI platform and a large language model (LLM) platform covering model development, training, inference, and serving.
The AI platform supports scenarios including Notebook, training, offline inference, online inference, and image building. Notebook environments need to mount user directories and datasets. Training and inference jobs need to access input data, models, and intermediate results. The image-building process also generates reusable caches.

The LLM platform aggregates different model services and provides model invocation and benchmarking capabilities. The data and reports generated during model benchmarking also need to be persistently stored.

For different types of data and workloads, we primarily use three types of storage:

CPFS provides high-performance parallel storage.
JuiceFS serves as the shared file storage foundation for the platform. It supports data requiring POSIX semantics, multi-node sharing, and a unified namespace, including Notebook environments, task datasets, model benchmarking reports, and image build caches.
Object storage is mainly used for object data such as multimedia attachments.

A small number of directories on development machines are still used as a transitional solution and have not yet been migrated to the unified storage system.

Overall, this storage architecture already covers the data persistence and cross-environment sharing requirements of our AI platform.

image: Hello's AI infrastructure
caption: Storage architecture in Hello's AI infrastructure

New storage requirements introduced by agent workloads

As agents have gradually moved into production, our AI capabilities have expanded beyond model development, training, and inference to agents capable of invoking tools and completing application workflows.
Related applications include image review for two-wheeled vehicle operations and trip recording review for ride-sharing services. These applications have delivered practical results by expanding review coverage, reducing manual effort, and lowering service costs.
Building on the deployment of vertical agents, we further introduced Jarvis Claw, a personal agent for employees. Employees can interact with Jarvis Claw through DingTalk or the web. By combining capabilities such as long-term memory, skills, Model Context Protocol (MCP), and Cron, it can assist with tasks such as answering questions, generating reports, reviewing content, and performing analysis.

Unlike vertical agents designed for a single application workflow, a personal agent needs to preserve state across multiple rounds of interaction and continuous tasks.

For example:

In an IT support scenario, Jarvis Claw needs to preserve user requests, attachments, conversations, knowledge retrieval results, and tool invocation results.
In an operations reporting scenario, it needs to continuously access historical source tables and scripts while generating new data tables, charts, and reports.
In an operations attribution scenario, it also needs to preserve data extraction scripts, intermediate data, and final analysis results for later inspection and re-execution.

A long-running agent needs to preserve five types of data:

User profiles and behavioral configurations
Conversation history and runtime state
Long-term memory that can be reused across conversations
Skills, MCP servers, channel configurations, and scheduled task configurations
Application artifacts such as scripts, CSV files, Excel spreadsheets, charts, and reports

These states naturally take the form of files:

Profiles are stored as Markdown files.
Conversation history is stored in SQLite.
Skills consist of directories and scripts.
Task configurations use JSON.
Application results exist as ordinary files such as data tables, charts, and code.

The complete state of an agent ultimately consists of a continuously evolving collection of files and directories that are created, modified, and related to one another.

A long-running agent therefore requires a unified shared storage layer that supports POSIX semantics, SQLite write-ahead logging (WAL), atomic rename operations, and access to large numbers of small files. It must also allow multiple Pods to share the same user state. Even if a compute Pod is destroyed, recreated, or rescheduled, the user's profile, conversations, memories, and task artifacts must remain available.

How JuiceFS stores the complete state of an agent

To meet these requirements, we extended our existing JuiceFS-based shared file storage foundation to agent workloads and use it to store a complete workspace for each user.

The platform assigns each employee an independent Jarvis Claw directory based on the employee account. The directory structure is similar to:

/userdata/aiplatform-jarvis-claw/user/<staff_id>/data/workspace

The user's profile, conversations, memories, skills, task configurations, and application artifacts are all stored in this directory. The Jarvis Claw Pod is responsible only for computation.

When a Pod is destroyed, recreated, or rescheduled, it only needs to remount the existing workspace to continue using the previous state. This makes the agent stateful while keeping the compute Pods stateless.

What agent state is stored in the workspace?

The workspace contains profile files such as:

AGENTS.md
SOUL.md
PROFILE.md
HEARTBEAT.md
MEMORY.md

These files describe the agent's identity, behavior, and user preferences and are combined into the system prompt.

Conversation and runtime state are primarily stored in files and directories such as:

sessions/
transcripts/
dialog/
tool_results/
history.db
chats.json
inbox_events.json

history.db uses SQLite and also generates WAL and SHM files.
User messages are written to transcripts/ at the beginning of a task. Even if the task is interrupted later, the conversation content generated before the interruption is preserved.
Long-term memory is stored in directories such as:

memory/
digest/
mem_agent/
mem_session/
mem_metadata/
resource/

Jarvis Claw manages memory using ReMe and a Memory-as-File approach.
Some memory is written directly by the agent during execution. Another part is generated by periodically reviewing historical conversations, extracting user preferences and information that remains useful over time, and writing the results to directories such as digest/. Internally, this process is referred to as Dream.
The workspace also stores task and capability configurations, including:

cron_jobs.json
cron_runs/
mcp.json
channels.json
dingtalk_webhooks.json
skills/
skill.json

Skill packages are stored as ZIP files in object storage. When a Pod starts, they are downloaded and extracted into the user's workspace.
Scripts, data tables, charts, and reports generated by the agent are stored in directories such as output/ and scripts/, as well as in files including Excel spreadsheets, CSV files, and SVG images.

The actual workspace also contains auxiliary content such as browser data, backups, and migration files.

Together, these files and directories form the complete state of an agent and are continuously created, modified, and accumulated as tasks are executed.

How a unified file system supports data movement and operations

Storing agent workspaces in JuiceFS not only provides state persistence and storage-compute separation, but also allows agents to share the same file system with environments such as Notebook, training, and evaluation.

When permissions allow it, Notebook can directly access an agent's workspace. Data prepared in Notebook can be directly provided to an agent. Likewise, data tables and scripts generated by an agent can be reused for analysis or evaluation without repeatedly uploading or copying data between different systems.

Operational tasks such as inspection, migration, backup, and per-user storage usage analysis can also be performed using standard tools such as ls, du, and rsync. Engineers can directly inspect conversations, memories, configurations, and task artifacts to troubleshoot issues during execution.

However, as the workspace evolves from an ordinary data directory into a working environment that an agent can actively read from, write to, and execute scripts within, the platform must provide not only file persistence and sharing, but also appropriate access controls and execution boundaries.

Security boundaries for shared workspaces

An agent may not only read and modify files in its workspace, but also execute commands or code through Shell, MCP, and REPL. Without effective constraints, its access could extend beyond the user's workspace to other users' data, service source code, and sensitive files inside the container.

Therefore, we established security boundaries across four layers:

User data isolation
Tool invocation governance
File access control
Subprocess management

How different users' data is isolated

We manage file system permissions through JuiceFS and OpenACL, assigning each user an independent workspace and controlling the directories that different users are allowed to mount and access.

With this mechanism, each agent can access only the workspace for which it has been authorized and cannot read other users' data.

File system permissions provide the basic data boundary between users, but they cannot cover all access behavior generated when an agent executes tools and code inside a container. Therefore, in addition to user-level permissions, the platform also needs to further restrict tool invocations and their execution environments.

How the system determines whether a tool invocation is allowed

When the model decides to invoke a tool, the operation is not executed immediately. Instead, the governance layer first evaluates the operation through assert_policy().
The result falls into one of four categories:

ALLOW: Allow the operation.
DENY: Reject unknown tools and dangerous operations such as sudo.
ASK: Request manual approval.
SANDBOX_FALLBACK: Execute the operation in a sandbox environment.

When manual approval is required, the system sends an approval request to the frontend through a side channel. After the user approves the request, a generalized version of the rule can be written to policy.yaml so that similar future invocations can reuse it.
For Shell operations, execute_shell_command() fixes the current working directory to the user's workspace rather than the service source directory. Then, it selects the appropriate sandbox configuration based on the governance result. The configuration prioritizes dynamically generated fine-grained rules from the governance layer, followed by globally configured default rules registered when the service starts. Bare-process execution is allowed only in local development environments. This layer determines whether a tool invocation is allowed to execute. For approved operations, the platform must further restrict the files that can actually be accessed based on the execution method.

How file access permissions are restricted for the main process and subprocesses

Agents access files mainly in two ways, which require different control mechanisms.

File tools such as read_file, write_file, edit, grep, glob, and view_image run directly inside the main service process and do not create subprocesses.

As a result, they cannot be intercepted by bubblewrap. The system uses application-level path checks, such as is_protected_path(), to constrain relative paths to the user's workspace while blocking access to protected locations such as /app, which contains service source code and installation files. Protected entries are also filtered during directory listing.

Shell commands, stdio MCP servers, and REPL environments used to execute model-generated code, on the other hand, create subprocesses.

For these operations, application-level path checks alone cannot establish a complete security boundary. Therefore, the system uses bubblewrap and Linux mount namespaces to construct an isolated file system view for subprocesses.

Within this restricted view:

The user's workspace is mounted with read and write access.
Other approved paths are exposed according to policy rules.
Service source directories such as /app are hidden or mounted as read-only.
Sensitive paths such as ~/.ssh and ~/.aws are blocked.

As a result, the agent can operate normally within the user's workspace without gaining access to all files in the container.

If the production environment is configured to use bubblewrap but the sandbox cannot actually be created, Shell and MCP use a fail-closed policy: execution is rejected rather than falling back to an unisolated bare process. The REPL follows the same rule when executing model-generated Python code. If the sandbox is unavailable, execution is denied.

How subprocess lifecycles and sandbox boundaries are managed

Each tool invocation that creates a subprocess uses an independent sandbox and process group with --die-with-parent enabled. When a user executes /stop, a task times out, or a new task within the same Scope replaces the current one, the system progressively cancels the current task and uses killpg to clean up the entire process group. This prevents Shell processes, MCP servers, or their child processes from continuing to run in the background.
Although stdio MCP servers are launched by the SDK itself, their execution commands are also prefixed with bubblewrap so that they use the same file system view as Shell commands. The REPL must likewise enter the sandbox when executing model-generated Python code, preventing execution methods other than Shell from becoming a path around the security boundary.
There are also several implementation details to consider:

After /app is overlaid with tmpfs, it must be remounted as read-only to prevent writes to the temporary /app from silently succeeding.
Before entering the sandbox, VIRTUAL_ENV and PATH entries pointing to protected locations must also be cleaned up to avoid problems with the Python runtime environment.
When the service starts, it performs an actual /bin/true execution inside the sandbox as a self-check rather than merely verifying that the bubblewrap binary exists.

The current solution primarily addresses file system and service source code isolation. Network isolation through --unshare-net has not yet been implemented. In addition, CPU and memory quotas have not yet been configured beyond execution timeouts.

Summary

Our production practice can be summarized in four points:

Everything about an agent's state can be represented as files. Profiles are Markdown files. History is stored in SQLite. Skills are directories. Memory and configuration are stored as files. Application artifacts include Excel spreadsheets, CSV files, charts, and scripts.
A shared file system provides state persistence and storage-compute separation. Agent Pods are responsible only for computation, while agent state and workspaces are stored in JuiceFS. Pods can be rebuilt at any time without causing user data to disappear with the compute instance.
The same file system connects multiple AI environments. Notebook, training, evaluation, and Jarvis Claw can access the same set of files. Data no longer needs to be repeatedly copied between different systems, and operations can rely on standard file-based tools and workflows.
Opening the workspace requires layered security boundaries. JuiceFS and OpenACL manage user-level file system permissions. Application-level path checks restrict the file access scope of tools running in the main process. Bubblewrap controls subprocesses created by Shell commands, MCP servers, and code execution.

Next, we plan to further explore agent self-evolution. Agent telemetry, memories, and evaluation data will continue to accumulate in the file system. Models will continue to improve, and this accumulated data will become part of the foundation for the agent's future evolution.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Scaling PB-Scale Scientific Data Workloads: How the American Museum of Natural History Powers Research Computing with JuiceFS

Thu, 03 Sep 2026 05:29:00 +0000

In cutting-edge scientific fields like astrophysics and genomics, high-throughput reading of massive datasets and zero-tolerance data integrity are the absolute lifebloods of computational analysis.

Recently, the JuiceFS team sat down for an in-depth conversation with the Scientific Computing team at the American Museum of Natural History (AMNH). They explained how they use JuiceFS Community Edition to handle large amounts of research data within an Open Storage Network (OSN) S3-compatible storage system which was funded by a grant from the National Science Foundation (NSF), creating a highly reliable and consistent storage setup for academic use.

The workload: 95% read-heavy traffic

The scientific computing environment at AMNH supports extraordinarily intense interdisciplinary research, operating at a massive scale. Currently, AMNH's OSN nodes have the capacity to store around 3 PB of research data, with approximately 30% of that total volume actively accessed via the NFS + JuiceFS layer. The overall storage architecture is precisely tailored to handle these specific academic patterns:

A 95% Read-Heavy Profile: Workloads include the system's massive reference datasets such as genomic sequences and astronomical survey data in read-only format. The calculations are very throughput- and latency-sensitive. Once a dataset is written, it will be read many times and rarely changed.
The OSN (Open Storage Network) Model: AMNH partners with the Open Storage Network. Standard rack servers and an external JBOD are hosted on-site at the museum but managed remotely by OSN administrators. This setup provides AMNH with cost-effective S3-compatible endpoints while allowing them to maintain physical ownership of the hardware and data.
Seamless Multi-Protocol Access: AMNH adopted JuiceFS Community Edition as the distributed file system layer, allowing mounting S3 buckets seamlessly and enabling traditional NFS access to this underlying S3-compatible storage for high-performance computing (HPC) clusters.

image: NFS + JuiceFS + OSN
caption: AMNH's NFS + JuiceFS + OSN storage architecture

Data integrity first, without silent corruption

For scientific research, data consistency is crucial. To prove and continuously observe the system's resilience, the AMNH team conducted rounds of tests during the PoC phase and watched JuiceFS logs and metrics closely in production.

"We have recently observed timeout warnings when JuiceFS writes to the OSN object store, but data integrity issues were never reported. We do want to understand why, though."

– Scientific Computing Team, AMNH

That was one of the discussions we had during the meeting. It turns out that JuiceFS configures all object store SDKs with RetryMaxAttempts=1 and implements custom logic for retries. This way, JuiceFS is able to unify the behavior across all object stores it supports. Thus, in this case, the warning is a precaution, and the data integrity is ensured.

When comparing JuiceFS to other file systems that claim higher raw performance metrics (often achieved by using "unsafe" write-handling or skipping strict metadata commits), AMNH opted for data consistency.

"We use almost all the default configurations for the JuiceFS client with Redis as the metadata engine, and it's been running in production steadily for 4 years now."

– Scientific Computing Team, AMNH

Current & future tuning options

The conversation also continued into how AMNH tunes and considers the JuiceFS client configurations to match their scientific computing patterns needed for the future.

The write-back configuration

Currently, the only non-default configuration AMNH utilizes is --writeback , which smooths out the client-side writing throughput. There are always tradeoffs. Write-back sacrifices a bit of data consistency and requires high disk reliability on the mounting machine. However, given AMNH's current workload pattern (mostly reads), both teams agreed that it is acceptable either way, write-back enabled or not.

High-performance caching

The teams also evaluated the differences between the Linux kernel's native page cache and JuiceFS' cache for client-side metadata. While the kernel page cache is the fastest, the JuiceFS client-side metadata cache offers more flexibility. In a read-heavy setup, this configuration can save many roundtrips to the metadata engine. Currently, the metadata engine is co-located with the JuiceFS client-mounting host, but in the future, this configuration can be a nice add-on for performance.

On the file side, since the hardware uses HDDs managed by the OSN object store, the JuiceFS client's local data read cache significantly lowers latency after the first read. Increasing the --cachesize can further improve performance when needed.

The potential of using SQLite as the metadata engine

In a highly creative approach, the teams discussed utilizing SQLite on local NVMe storage as their JuiceFS metadata engine. The JuiceFS team highlighted that for standalone sandboxes or read-heavy workloads, SQLite is an exceptional, low-maintenance choice, provided the metadata file is backed up for resilience (which is true for Redis as well). Since for AMNH's current scenario, the primary I/O bottleneck usually lies in the backend object storage rather than the database, SQLite's simplicity makes it incredibly attractive for specific workloads.

Final thoughts

Managing multi-petabyte scale, strong data consistency, and high-throughput research workloads is a difficult balancing act for any infrastructure team. AMNH has built a resilient, low-maintenance storage environment that lets scientists work without creating massive engineering overhead, using an open architecture and JuiceFS Community Edition on top of an OSN backend.

Reflecting on their four-year journey of running this system stably in production, the AMNH team summarized their experience seamlessly:

"The implementation on our side is straightforward, and we are happy about it overall."

– Scientific Computing Team, AMNH

As data demands in scientific computing continue to grow, this blueprint of combining cost-effective object storage with a robust, reliable file system layer provides a highly reproducible success story for academic and research institutions worldwide.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and our Discord server.

How Tencent Cloud Built a Unified Storage Platform with JuiceFS and FoundationDB

Thu, 27 Aug 2026 08:39:00 +0000

As big data and AI workloads continue to grow, more enterprises are transitioning from tightly coupled storage-compute architectures to storage-compute separation architectures. However, in real-world deployments, enterprises often need to manage multiple storage systems simultaneously, including Cloud Object Storage (COS), S3, and HDFS. These backends differ in access protocols, authentication mechanisms, file system semantics, and metadata performance. This creates challenges in unified access, existing data reuse, and cache acceleration.

To address these challenges, the Tencent Cloud (a cloud computing service operated by Tencent, a multinational technology company headquartered in China) team built an enterprise-grade unified storage solution based on JuiceFS+FoundationDB. The solution provides full POSIX semantics for newly created file systems while also integrating existing object storage and HDFS data into a unified namespace, enabling centralized access and cache acceleration.

In this architecture, FoundationDB serves as the metadata engine, supporting metadata management at the tens-of-billions scale with strong transactional consistency. This is also the first time the JuiceFS community has shared a production practice using FoundationDB for metadata management. Local caching and distributed caching are introduced to reduce remote access latency. Going forward, our team plans to further evolve the platform in areas including lakehouse integration, AI training acceleration, intelligent data tiering, and cross-cluster federation.

Challenges: storage gets more complex after storage-compute separation

During early customer discussions and proof-of-concept (POC) projects, we found that when enterprises migrate from storage-compute integrated architectures to storage-compute separation architectures, their biggest concerns are not the architecture model itself, but two questions:

Can existing data and applications be migrated smoothly?
Can performance remain close to the original integrated architecture after migration?

Enterprises often operate multiple storage systems simultaneously, including COS, S3, and HDFS. These backends differ in access protocols, SDKs, authentication methods, and file system semantics. If Spark, Hive, Flink, or AI training frameworks need to integrate with each storage system separately, development and maintenance costs can quickly increase.

Performance is another major challenge in adopting storage-compute separation. When data access paths become longer, remote storage and network communication introduce additional overhead. Meanwhile, running multiple storage systems independently makes it difficult to share and reuse caching capabilities.

Therefore, a unified storage solution must not only solve multi-backend integration challenges, but also provide reusable cache acceleration capabilities.

In addition, although object storage is well suited for massive-scale data storage, it’s generally less efficient than traditional file systems for metadata-intensive operations such as:

List
Stat
Directory traversal
Small-file access

Different storage backends also provide inconsistent support for file system semantics such as directories, permissions, quotas, and snapshots. These differences further complicate unified access for big data and AI workloads.

Based on these requirements, we chose JuiceFS as the foundation for building an enterprise-grade unified storage platform.

Architecture: unified access with two operating modes

JuiceFS adopts a metadata-data separation architecture. The metadata engine can be selected based on workload scale and consistency requirements, including Redis, TiKV, and other options.

For this solution, we selected FoundationDB as the metadata engine, mainly because of its:

Strong transaction model
Ordered key-value data model
Automatic data distribution
Horizontal scalability

These capabilities enable metadata management at the tens-of-billions scale.

Based on JuiceFS' core architecture, we built a unified storage platform consisting of:

A unified client layer
A service-oriented metadata layer
Multi-level caching
Multi-backend storage adapters

JuiceFS provides fundamental capabilities including:

File system semantics
Multi-protocol access
Object storage-based data paths
Local caching

On top of these capabilities, we further extended the system with the following capabilities to meet the multi-tenant and multi-storage requirements of cloud service scenarios:

Unified authentication
Multi-volume management
Unified namespace
Existing data integration
Distributed caching

On the client side, big data and AI applications can access the system through multiple interfaces:

HDFS interface
POSIX mount
S3 interface
Python SDK

The client supports:

Hadoop SDK
Python SDK
FUSE mount
Local cache
Service discovery

This allows frameworks such as Spark, Hive, Flink, Presto, and various AI training frameworks to access storage without adapting individually to different backend systems.

image: JuiceFS+FoundationDB
caption: Unified storage architecture based on JuiceFS+FoundationDB

For metadata access, we converted JuiceFS' metadata access path into a service-oriented architecture.

Clients no longer connect directly to FoundationDB. Instead, they communicate with stateless metadata services through RPC. The metadata service centrally provides:

Metadata APIs
Authentication and authorization
Multi-volume management
Quota management
Directory protection
File protection
Task management

ZooKeeper is used for:

Service registration
Service discovery
Cross-node state synchronization

The metadata service can scale horizontally as needed, while FoundationDB acts as the backend metadata engine responsible for metadata persistence and transaction processing.

image: Data access architecture
caption: Data access architecture for managed mode and external mode

To cover both new file system creation and existing data ingestion scenarios, the underlying storage layer provides two modes:

Managed mode is designed for newly created file systems. File metadata is written to FoundationDB through the metadata service, and file data is split according to JuiceFS' chunk/slice/block model and written to object storage. This provides full POSIX file system semantics.
External mode is designed for existing data reuse. Its core is the Unified File System (UFS) abstraction layer, which abstracts backends such as COS, S3, and HDFS into a unified file system interface, without migrating or taking over the existing metadata and data of those backends. Metadata requests are passed through the metadata service via UFS to the underlying storage, while data reads and writes are performed using the native protocols of the respective backends.

image: UFS multi‑storage adaptation architecture
caption: UFS multi‑storage adaptation architecture in external mode

The unified namespace organizes both modes within a single directory structure. To upper‑layer applications, it presents a unified path and access entry. As for the underlying implementation, different directories can correspond either to managed file systems or to mounted existing HDFS or object storage data. This allows new and existing data to coexist under the same access system without requiring large‑scale data migration beforehand.

On the data access path, the system provides both local caching and distributed caching. Read requests first check the local cache; if missed, they access the distributed cache; if still missed, they fall back to the underlying storage. This reduces remote access latency and backend storage pressure.

FoundationDB: design trade‑offs for 10‑billion‑level metadata

In the unified storage access layer, the metadata engine determines the system's scalability limit and consistency capabilities. When file counts reach the billion or even 10‑billion level, the metadata system itself becomes the critical bottleneck.

Why we chose FoundationDB

During the metadata engine selection phase, we focused on several key questions:

Whether a single cluster can support 10‑billion‑level metadata
Whether the transaction model is strong enough
Whether automatic sharding and horizontal scaling are supported
Whether operational complexity is manageable

Traditional relational databases tend to hit bottlenecks in single‑volume scale and horizontal scalability, so we focused our evaluation on distributed key-value solutions, including TiKV and FoundationDB:

TiKV has been widely adopted in large‑scale metadata scenarios and offers strong horizontal scalability.
FoundationDB better meets our requirements in terms of strict transaction consistency, ordered key‑value model, automatic data distribution, multi‑replica strong consistency, and operational complexity.

Ultimately, we chose FoundationDB as the metadata engine because it’s better suited for the strongly consistent, high‑concurrency, range‑scannable access patterns typical of file system metadata. This practice also allowed our team to accumulate experience in modeling, tuning, and operating FoundationDB in large‑scale metadata scenarios, providing a reference for future use cases such as Hive table metadata management.

Key design: multi‑volume isolation and range scans

File system metadata naturally has two types of access patterns:

A single cluster must host multiple file systems, requiring clear boundaries between volumes.
Operations such as directory traversal, attribute queries, and chunk index queries often rely on prefix scans.

Therefore, key design must both satisfy multi‑volume isolation and closely align with file system access paths, avoiding hot spots or large‑transaction issues at scale.

Leveraging FoundationDB's ordered key-values, we used JuiceFS metadata design method:

Using fsname as the key prefix to distinguish different file systems
Organizing attributes, directory entries, chunk indexes, and other metadata related to the same file within a close key space to improve access locality
Encoding numeric fields in big‑endian order so that lexicographic order aligns with numeric order for sequential scans.

image: FoundationDB metadata key
caption: FoundationDB metadata key encoding design

On this basis, multiple file systems can share the same FoundationDB cluster while maintaining independent key spaces. For operations such as list that may scan large amounts of data, pagination is used to control single‑transaction size. In transaction conflict scenarios, optimistic locking and automatic retries are used to reduce lock contention.

How to avoid FoundationDB's hard limits

FoundationDB has explicit hard limits on key size, value size, and transaction size:

A single key is limited to 10 KB.
A single value to 100 KB.
A single transaction to 10 MB total size.

These limits cannot be adjusted via configuration parameters. When using FoundationDB as a file system metadata engine, we need to pay special attention to operations that tend to enlarge metadata size, such as frequent random writes, repeated truncates, fallocate punch hole, and copyFileRange, which may cause individual values to grow continuously. Large‑scale file copies, large file truncation, batch metadata import, or deleting an entire file system may trigger oversized transactions.

The highest risk comes from chunk slice accumulation. In JuiceFS' data model, the slice list corresponding to a chunk is stored in a single value, and each slice occupies about 24 bytes. When a chunk accumulates about 4,266 slices, it may hit FoundationDB's 100 KB value limit. Operations such as frequent random writes to the same chunk, repeated truncates, fallocate punch hole, and copyFileRange will continuously increase the slice count. Without a governance mechanism, a single value can keep growing, eventually causing transaction commit failures.

This issue also has a subtle implementation aspect: some paths use atomic append operations like AppendIfFits. This is a blind‑write operation; the transaction cannot know the total value size after the append in advance. If the append exceeds 100 KB, failure may not occur until the transaction commit phase. This means the problem is not exposed early during writes but instead accumulates silently until a commit fails. Therefore, relying solely on retries after failure is insufficient; safety boundaries must be set in the data model and write path.

To address chunk slice accumulation, we implemented and strengthened the compaction mechanism. The core idea is to detect chunks with too many slices during read/write operations and merge multiple small slices into fewer large slices to control individual value growth. For lightly fragmented chunks, asynchronous compaction can be triggered to avoid blocking writes. When the slice count reaches a safety threshold, synchronous compaction is triggered to prevent it from growing to the 100 KB limit. In practice, maxSlices is set to 2,500, corresponding to about 60 KB, leaving about 40 KB of headroom below the 100 KB limit.

During compaction implementation, concurrency and reclamation issues also need to be handled. The merge process uses CAS semantics to ensure the chunk has not been modified concurrently, avoiding data loss. If compaction fails, normal writes are not affected, and it can be retried later. Obsolete slices generated after compaction are passed to the Garbage Collection (GC) process, where reference counting determines whether they are still in use. A delayed‑deletion mechanism supports accidental deletion recovery and background reclamation.

In addition to chunk slice accumulation, we also examined other risk items such as file system configurations, Kerberos tokens, POSIX locks, extended attributes, and UFS path‑type keys. Overall, these risks are manageable: most values are small, and key design is constrained by file name or path length. The real area requiring governance is still the single‑value expansion caused by chunk slice accumulation.

Cache acceleration system: two‑level caching to reduce remote access overhead

After storage‑compute separation, data access paths shift from local disks to remote object storage or HDFS. For read‑intensive workloads such as big data analytics and AI training, if every read goes directly back to the underlying storage, network latency and backend access pressure are amplified. Therefore, the unified storage layer must not only address "how to access multiple storage types" but also "how to make remote data reads faster."

Our approach is to introduce two‑level caching: client local cache as L1, and distributed cache as L2. Read requests first check the local cache; if hit, return directly. If missed, they access the distributed cache. If still missed, they fall back to the underlying storage and fill the data into the cache system. This provides both the lowest latency from local SSD or memory and the ability to reuse hot data across multiple clients through an independent SSD cache cluster.

Local caching mainly uses JuiceFS Community Edition's existing capabilities, extended to support external mode. It runs inside the client process, caches data at block granularity by default, and supports least recently used (LRU) eviction, sequential read prefetching, and OS cache control. The distributed cache is our self‑developed independent cache cluster. Clients use consistent hashing to route the same data block to the same cache node, making it suitable for multi‑node shared reads, scenarios with limited client local storage, or workloads requiring cross‑task hot data reuse.

The most critical area in the cache system is consistency. In managed mode, every write generates a new slice ID. This causes the cache key to change, so old caches are naturally not hit. Meanwhile, objects in object storage are never modified in place after being written, so no additional invalidation mechanism is needed.

External mode is more complex, because files in the underlying HDFS or object storage may be directly overwritten by external systems. If the cache key contains only the path, stale data may be read. To address this, we introduced a fingerprint mechanism in external mode, incorporating file path, block offset, and file fingerprint into the cache key. The fingerprint is composed of file modification time and size, and is frozen at open time. When the file is modified externally, the fingerprint changes, so old caches are naturally not hit, satisfying close‑to‑open consistency.

This brings a trade‑off: managed mode provides finer‑grained cache isolation based on slice IDs, while external mode relies on file fingerprints, which are more coarse‑grained and may invalidate larger cache ranges. However, for scenarios where existing data may be modified by external systems, this is a necessary trade‑off for consistency, and it’s also an area for future optimization.

In addition to basic read caching, the system supports cache warm-up, cache‑stampede protection, TTL‑based eviction, and transparent degradation. For example:

The warmup command can pre‑load hot data.
Distributed caching supports batch-warming by directory prefix.
When the cache layer becomes unavailable, the system can automatically degrade to direct underlying storage access to prevent cache failures from affecting application reads.

Multi‑tenancy and elastic scaling

In cloud provider scenarios, the unified storage layer must serve multiple business units, tenants, and file systems simultaneously. Therefore, we designed the metadata service as a stateless architecture. Clients perform service discovery and load balancing through ZooKeeper, and metadata service nodes can scale horizontally on demand to increase overall access capacity and availability.

Multi‑volume isolation relies on FoundationDB's key prefix design. Each file system uses an independent fsname prefix, and different volumes have independent key spaces, allowing a single FoundationDB cluster to host multiple file systems. New file systems can be added online without restarting the metadata service.

image: Stateless metadata
caption: Stateless metadata service and FoundationDB multi‑volume isolation architecture

In multi‑node deployments, runtime information such as quotas, permissions, directory protection, and task states must also be synchronized across nodes. We use ZooKeeper for cross‑node change notifications, combined with debounce merging, precise key‑level refresh, and periodic polling for fallback, ensuring configuration changes are promptly synchronized to all metadata service nodes.

Through stateless metadata services, fsname‑based isolation, and ZooKeeper state synchronization, the system can achieve both multi‑tenant isolation and horizontal scalability at both the service and metadata layers.

10‑billion‑level metadata validation

We conducted 10‑billion‑level metadata tests based on JuiceFS 1.3.0 and FoundationDB 7.3. The test environment comprised 6 x86 servers running Kylin V10 SP3, with FoundationDB deployed using triple replication and the SSD storage engine.

The tests were performed after 10.8 billion metadata entries had already been written. Results showed that at the 10‑billion scale, FoundationDB single‑operation latency remained largely consistent with baseline data. Some operations, such as readdir_1k and lookup, even performed better. Overall, after reaching 10.8 billion inodes in a single volume, metadata operation latency remained in the sub‑millisecond to 2 ms range, indicating no significant performance degradation at scale.

The table below shows benchmark single‑operation latency (μs/op, lower is better):

Operation	Redis-always	FoundationDB (official)	Current environment (10B+)	vs. Redis
mkdir	558	1,842	1,763	3.2x
create	570	1,761	1,720	3.0x
rename	728	1,911	1,849	2.5x
unlink	658	1,940	1,887	2.9x
lookup	173	1,029	817	4.7x
getattr	87	504	414	4.8x
write	553	1,747	1,710	3.1x
readdir_10	280	1,744	1,357	4.8x
readdir_1k	1,490	15,276	1,185	0.8x

In extreme write tests, we used 15 concurrent processes to perform juicefs clone for metadata cloning, validating FoundationDB's write ceiling. Results showed that under the 6‑node + SSD configuration, stable‑phase write throughput was approximately 11,000–15,000 inodes/s. Bottleneck analysis indicated that the main pressure was on storage write queues and disk I/O, rather than CPU or network. Future improvements can be made by adding storage processes, upgrading disk performance, or scaling Commit Proxy and TLog components.

Inode creation:

Phase	Inode creation rate	Description
Initial peak	23,000+ inodes/s	Peak performance when the write queue is not yet full
Stable phase (after 5–6 minutes)	15,000 inodes/s	Write queue gradually fills up
After queue full (after 10+ minutes)	11,000–13,000 inodes/s	Disk I/O becomes the bottleneck; FDB throttling protection is triggered

For high availability, we validated scenarios with 1‑node and 2‑node failures. With 1‑node failure, the service remained available, and FoundationDB automatically triggered replica repair and data migration. With 2‑node failure, the service remained available, but the cluster's fault tolerance dropped to zero—it could no longer tolerate additional failures. After recovery, the system triggered data rebalancing, with a complete recovery time of approximately 4 hours in testing. This process introduces additional I/O pressure, so production environments must pay close attention to disk watermarks, single‑disk failures, and resource consumption during recovery.

For capacity planning, test data showed that at 5 billion inodes, FoundationDB disk usage was approximately 6.1 TB; at 10.8 billion inodes, it was approximately 10.1 TB. After accounting for triple replication, compression, and reserved space, approximately 120 GB per 1 billion inodes can be used as a baseline capacity estimate.

Through this validation, we’ve essentially confirmed that FoundationDB can support 10‑billion‑level metadata per volume while maintaining stable metadata access performance at that scale. At the same time, the triple‑replica architecture provides good failure recovery capabilities, though production environments still require careful capacity planning, disk monitoring, and I/O management during recovery.

Summary and future plans

At present, this unified storage access layer built on JuiceFS+FoundationDB has completed core capability construction in big data storage‑compute separation scenarios, including unified client access, leveraging of external data, multi‑level cache acceleration, 10‑billion‑level metadata management, and multi‑tenant scalability. Moving forward, we’ll continue to evolve around lakehouse integration, AI training, and storage governance.

In the lakehouse direction, we plan to strengthen integration with table formats such as Iceberg and Hudi through the unified storage layer, enabling upper‑layer data lake and data warehouse scenarios to reuse unified storage access, permission management, and cache acceleration capabilities.

For AI training, we’ll further optimize cache warm-up and near‑local caching to reduce remote read amplification and GPU wait times in training jobs. We’ll also explore model checkpoint management using snapshot capabilities, making intermediate state saving, recovery, and reuse more efficient during training.

In storage governance, we’ll explore intelligent tiering capabilities, automatically migrating data between SSD, HDD, and object storage based on access frequency, achieving a better balance between performance and cost. For larger‑scale cloud deployments, we’ll continue to advance cross‑cluster federation capabilities, enabling metadata synchronization, routing, and unified access across multiple clusters.

Going forward, we’ll continue to iterate around performance, cost, elasticity, and data governance, enabling more workloads to access and manage underlying data in a unified manner.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

JuiceFS Blog

How Zhuoyu Built a Petabyte-Scale Data Platform for Autonomous Driving with JuiceFS

The autonomous driving data loop: growing volumes and repeated data consumption

Coupling between access, operations, and scaling in the original architecture

Architecture upgrade: building a unified data platform on JuiceFS

Technology selection: S3 as the primary interface, JuiceFS as the storage foundation

Unified access layer: decoupling applications from storage

Building the core capabilities

Multiple storage pools: scaling capacity and isolating failures

Distributed caching: reusing hot data across nodes

UFS: integrating external data and accelerating access

Production at hundreds-of-petabytes scale: architecture and optimizations

Cache orchestration: building caches as data arrives and proactively warming hot data

Inference workloads: large-scale model delivery

Training workloads: TCP zero-copy optimization

Lance columnar storage: reducing I/O at the data format layer

Combining Lance with JuiceFS for targeted data warm-up

Conclusion

Scaling Data Management for Agent Sandboxes with JuiceFS

How to manage long-lived data in short-lived sandboxes

How JuiceFS manages and accesses sandbox data

Organizing shared data and task data with directories

How cross-cloud sandboxes access the same data

JuiceFS data mounting mode for strongly isolated sandboxes

JuiceFS in cloud provider sandboxes

Tencent Cloud: data mounting methods in agent runtime

Alibaba Cloud: FC and ACS

Sandbox lifecycle management and mount control

Challenges and optimization in large-scale agent sandbox scenarios

Client scaling bottleneck

Cache reuse and data loading efficiency

Small files and metadata performance

Fault isolation and platform governance

How to Access JuiceFS via S3 and WebDAV Protocols

A shared initialization path

Applicable scenarios and common mappings

S3 Gateway principles: from object semantics to file system operations

Request path: from the MinIO S3 server to jfsObjects

State storage: how the internal namespace carries the Gateway state

Semantic conversion: PutObject, Multipart, Bucket, and Metadata

Permission coordination: IAM and multiple instances

WebDAV principles: from file protocol to JuiceFS operations

Request path: from HTTP entry to webdavFS

State boundaries: dead properties and LOCK

Nydus + JuiceFS: Reducing Container Startup Time for AI Inference from 116s to 1.4s

Why OCI image distribution is slow

Dragonfly+Nydus: what it solves and what it leaves behind

Architecture redesign: Nydus blobs hosted by JuiceFS

Runtime mechanism: how Nydus and JuiceFS coordinate

Startup phase: fetching bootstrap (metadata)

Read phase: on‑demand blob access

Cache phase: data reuse via JuiceFS cache groups

Production test data

Container startup time comparison

On‑demand file read performance

Summary: combining on‑demand loading with shared cache

JuiceFS Enterprise 5.4: From 100 Billion Files to 1 Million Clients

Supporting 1 million clients

Fast cloning for large directories

Performance improvements

More than 90% RDMA bandwidth utilization

Higher random-read IOPS under high concurrency

Optimizing data access across regions

On-demand synchronization to reduce data replication

Accelerating metadata access with read-only nodes

More flexible cache management

Improving hot-data cache hit rates

Preserving cache hit rates when adding multiple cache nodes

Reducing the number of files on large cache disks

Warming up specific byte ranges

Simplifying operations for massive files

Restoring files by deletion time

Reducing GC and fsck memory usage

Restricting access with token allowlists

Summary: scale changes everything

Infinite Workspaces for AI Agents: A Practical Architecture with JuiceFS, SQLite, and Litestream

The creative trio: JuiceFS + SQLite + Litestream

An unconventional but effective combo

The performance reality

Final thoughts

Request path: from the MinIO S3 server to `jfsObjects`

Semantic conversion: `PutObject`, `Multipart`, `Bucket`, and `Metadata`

Request path: from HTTP entry to `webdavFS`

Reducing GC and `fsck` memory usage