Founded in 2016, Zhuoyu Technology (ZYT) focuses on the research, development, and large-scale application of mobile intelligence technologies. Leveraging our proprietary native multimodal foundation models and generative AI technologies, we build a unified mobile intelligence technology base that can efficiently adapt to multiple scenarios. It not only provides automakers with open, customizable, production-oriented intelligent driving assistance solutions (covering L2 and above), but also delivers underlying technology for autonomous mobile robots in diverse scenarios, empowering a full-spectrum intelligent mobility ecosystem from the ground to near-space.
As R&D scale expands, training and inference compute are gradually moving toward multi-data-center and hybrid-cloud environments, while data volume and access demands continue to grow. Compute resources can be scheduled on demand, but task startup can still be affected by mount status, origin data fetches, and model loading. How to make data ready in time and reduce compute waiting has become a key goal of storage architecture evolution.
We built a unified data foundation on JuiceFS. Through unified access, multiple storage pools, and shared caching, it supports data ingestion, capacity scaling, and hot data reuse. Combined with proactive warm-up and read optimization, it accelerates data delivery. Today, the online cluster carries 100-PB-scale data, with online read capability reaching the 100 GB/s range. This article covers the architecture evolution and production practices.
The autonomous driving data loop: growing volumes and repeated data consumption
Autonomous driving R&D revolves around a continuous data loop. Data collected by road-testing and production vehicles is filtered and labeled before entering model training. The resulting models then go through evaluation and deployment, while newly identified issues and newly collected data feed into the next iteration. At each stage, how quickly data becomes available directly affects whether compute resources have to wait and, ultimately, how efficiently R&D teams can iterate.
From a storage perspective, autonomous driving R&D is a data-intensive pipeline. Multimodal data continuously flows back into the platform and is repeatedly consumed by different R&D workloads. Data collected by production and road-testing vehicles, together with simulation-generated data, first enters shared multimodal storage. Downstream workloads—including data filtering, labeling, model training, simulation evaluation, and individual development environments—then consume the data.
As data volumes and iteration frequency increase, storage must support both continuous capacity growth and efficient data access to keep downstream workloads supplied with data.
Coupling between access, operations, and scaling in the original architecture
Before adopting JuiceFS, we used both S3 APIs and file-based access. Multiple access paths coexisted, storage configuration was scattered across applications, and compute-side teams were responsible for maintaining mount clients.
We identified three core challenges:
- Mounting coupled with workload scheduling: Task startup depended on mount clients being ready. Maintaining those clients on the compute side increased operational overhead and could leave compute resources waiting for storage.
- Storage configuration coupled with applications: Storage details such as buckets, endpoints, and credentials were embedded in application code. Changes to underlying storage resources required application-side changes.
- Capacity coupled with bandwidth scaling: Data volumes continued to grow while workloads repeatedly accessed hotspot data. Local caches could not be reused across nodes, making it difficult to scale storage capacity and read bandwidth independently.
Therefore, the new architecture needed to simplify application integration and mount management while allowing storage capacity and read bandwidth to scale independently.
Architecture upgrade: building a unified data platform on JuiceFS
Technology selection: S3 as the primary interface, JuiceFS as the storage foundation
To maintain compatibility with existing workloads while supporting future growth, we chose JuiceFS as the storage foundation for four main reasons:
- Multiple access protocols: JuiceFS supports interfaces such as S3 and POSIX, allowing different workloads to use the access method that best fits their needs.
- Capacity scaling: JuiceFS uses object storage as the underlying data store, providing room for continued data growth.
- Cache acceleration: The Community Edition already provides local caching, giving us a foundation for further read-path optimizations.
- Open source ecosystem: An active community makes it easier to continue improving the platform based on production requirements.
Unified access layer: decoupling applications from storage
In the new architecture, applications access storage through an SDK provided by our internal data platform. The SDK provides a unified logical namespace and handles the mapping between logical paths and underlying storage locations and credentials. Applications integrated through the SDK primarily use S3 access, reducing the need to manage mount clients on compute nodes. Existing workloads can continue to use the POSIX interface.
Using S3 as the primary access path also makes it easier to implement traffic isolation, monitoring, and auditing at the access layer.
The unified access layer separates application changes from storage changes, reducing the impact of underlying resource changes on application integration. We can continue scaling capacity, adding caches, and integrating external data sources while keeping the interfaces exposed to applications stable.
Building the core capabilities
We extended the JuiceFS Community Edition with multiple storage pools, distributed caching, and Under File System (UFS). The resulting architecture is shown below.
Multiple storage pools: scaling capacity and isolating failures
We added routing to the data layer of a JuiceFS volume. Based on policies such as file size and configured weights, the router distributes writes across different storage pools. The backends can be multiple S3 buckets or file storage systems.
This gives us a one-to-many relationship between a JuiceFS volume and its underlying buckets. When an object storage resource reaches its scaling limit, we can add another storage pool to handle subsequent writes without migrating the existing data as a whole. Multiple independent storage pools also help reduce the impact of failures within any single resource domain.
Distributed caching: reusing hot data across nodes
Initially, we used local caching, which prevented nodes from sharing cached data. When a workload was scheduled to a new node and the required data was not available locally, the node had to fetch the data from the source again and keep another copy in its local cache. This increased read latency and reduced cache utilization.
To address this, we developed distributed caching of the Community Edition. Multiple clients can reuse hot data in a shared cache, while object storage remains the authoritative source of data. The read path works as follows:
- The client uses consistent hashing to select a cache node.
- If the data is available on the cache node's local disk, the node returns it directly.
- If the data is not cached and fetching from the source is allowed, the cache node reads it from S3, returns the data to the client, and keeps a copy locally.
Local client-side caching remains available, giving us a multi-level caching architecture together with distributed caching.
UFS: integrating external data and accelerating access
We also developed UFS of the Community Edition to bring external data sources into the JuiceFS namespace. UFS provides unified access to external data and accelerates reads through caching, while the authoritative copy remains in the original system.
For example, suppose an existing dataset is stored in S3 and we want to accelerate access using the JuiceFS cache. We can map the dataset to a path in JuiceFS without migrating the original data. Then, applications can access the data through the mapped path.
The current implementation uses read-only mappings. It supports the same backend storage types as the Community Edition. Metadata can be loaded on demand or warmed up in advance. Even without warm-up, JuiceFS can retrieve directory and file metadata from the external storage as needed when a directory is accessed.
Production at hundreds-of-petabytes scale: architecture and optimizations
Our production clusters now manage hundreds of petabytes of data, deliver hundreds of GB/s of bandwidth, and provide petabyte-scale cache capacity. To improve data preparation and workload read performance, we optimized cache orchestration, model delivery, network transport, and data formats.
Cache orchestration: building caches as data arrives and proactively warming hot data
Based on statistics collected through our SDK, we found that roughly 80% of the data is accessed within eight days after it arrives. Therefore, we build distributed caches asynchronously as data is ingested and plan cache capacity and TTLs based on access patterns, so that workloads can read data from the cache. Cached data is reclaimed when it expires or when cache capacity becomes insufficient.
For workloads that need explicit control over data preparation, we introduced Fluid for proactive orchestration. A Dataset describes the dataset and its source, while applications access the data through the corresponding Kubernetes PVC.
As part of the workload workflow, DataLoad works with the JuiceFS Runtime to warm the data. The application starts consuming the data only after it’s ready. The Fluid Controller coordinates resources and state throughout the process, managing the lifecycle from cache creation and application consumption to cache reclamation. For workloads that are not integrated with Fluid, we also provide APIs that allow applications to control cache warming and eviction themselves.
Inference workloads: large-scale model delivery
For inference workloads, data preparation primarily means delivering models. An inference Pod cannot start serving requests until its model has been loaded. Previously, Pods loaded models directly from the model repository at startup. As the number of nodes increased, the repository had to handle a large number of repeated reads. Therefore, we placed a cache layer between the model repository and inference nodes.
With this cache layer, models are first warmed up into the distributed cache and then distributed over the cache network to the local caches on inference nodes. Requests for the same data block are routed to the same cache node through consistent hashing, allowing the cache layer to aggregate I/O and reduce repeated reads from the source.
When multiple inference Pods load the same model, the shared cache can aggregate their read requests and keep the amount of data fetched from the model repository close to the size of a single copy of the model. In our internal inference workloads, this approach improved deployment speed by more than 10× compared with the previous architecture.
Training workloads: TCP zero-copy optimization
Once data is in the cache, workloads still need to continuously read it over the network. To reduce the overhead of this path, we optimized data transfer between clients and cache nodes, starting with TCP because it applies to a broad range of environments.
In the conventional read path, ReadAt copies data from kernel space into a user-space buffer, and then conn.Write copies it back into the kernel for transmission. When the requested data is available in cacheFile on the server and the requested range is valid, we can use io.CopyN → sendfile(2) to send the file data directly to the TCP socket. This eliminates this user-space copy.
To evaluate the performance of the optimized cache path, we ran a 1 MiB sequential-read benchmark. The combined bandwidth across multiple servers was 1,200 Gbit/s. As client load increased, measured throughput reached approximately 135–138 GiB/s. At around 135 GiB/s, P99 latency for 1 MiB requests was approximately 2 ms. When throughput increased further to around 138 GiB/s, P99 latency rose to approximately 20–30 ms. This shows that as throughput approaches the bandwidth ceiling, a trade-off remains between throughput and tail latency.
For the throughput-oriented MLPerf Storage UNet3D workload, we first used a high-density single-client topology. We concentrated the concurrent read load from eight accelerator ranks on a single node with a single 200 Gbit/s data ingress, allowing us to measure the data delivery ceiling of a single client. The tests covered three accelerator configurations: H20, H100, and B200. The H100 and B200 tests reached the limits of the 200G network and the client read path, shifting the bottleneck from backend storage to the client data ingress. In the next phase, we’ll extend the UNet3D training workload to multiple clients and combine it with RDMA to further evaluate sustained data delivery and accelerator utilization for large-scale, multi-node, multi-accelerator training.
Lance columnar storage: reducing I/O at the data format layer
In addition to improving storage-path throughput, we also reduced the amount of data and number of I/O operations required by workloads by changing the data format.
For example, consider labeling data. A single asset may correspond to multiple JSON files, which were previously packaged into tar archives. Downstream processing still had to parse the JSON frame by frame, resulting in repeated serialization and format conversion and generating a large number of small I/O operations.
We switched the labeling data to Lance and flattened commonly used JSON fields into columns. Data is passed in memory through Arrow batches and written to Lance tables in batches. Downstream workloads select only the required columns and read them in batches, reducing unnecessary reads, per-frame JSON processing, and repeated serialization.
The benefits vary across pipelines. I/O time decreased by approximately 20%–90%, IOPS requirements decreased by approximately 90%, and bandwidth consumption decreased by more than 60%. These improvements significantly reduced the demand that application workloads placed on storage resources.
Combining Lance with JuiceFS for targeted data warm-up
Column-based reads also allow us to make data warm-up more targeted. We use layout metadata such as the Lance Manifest to locate the physical byte ranges of the selected columns. Then, we map those ranges to the corresponding JuiceFS data blocks and warm only the required blocks, reducing unnecessary reads.
These capabilities have been merged into JuiceFS Community Edition: byte-range data warm-up (PR #7398) and a tool for parsing Lance datasets (PR #7399). Together, they can generate a list of files and byte ranges based on a specified dataset version and selected columns, which can then be used for targeted data warm-up.
Conclusion
Looking back at this evolution, we built a unified data platform on JuiceFS and gradually improved it along three paths:
- A stable access layer: The unified access layer abstracts differences between underlying storage systems. This minimizes the impact of storage scaling and deployment changes on application integration.
- Shared hot data: Distributed caching allows data to be reused across nodes. This reduces repeated reads from the source when workloads move between nodes, clusters scale out, or multiple copies of the same data are accessed.
- Proactive data preparation: Fluid orchestration and APIs allow data to be prepared before workloads consume it. This connects workload execution to the state of data preparation.
Together, these three approaches support data delivery at the hundreds-of-petabytes scale and performance requirements of hundreds of GB/s and above, while reducing the operational overhead and coordination required between storage and application teams. We continue to optimize cache orchestration, network transport, and data formats to make data available faster and reduce compute wait time, providing a more reliable storage foundation for autonomous driving R&D.
If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.
