Experience the Power of JuiceFS 1.1 Beta: Smoother, Convenient, and Efficient

We are excited to announce the release of JuiceFS 1.1 Beta! This version offers a variety of new functionalities and enhancements. It aims to improve your experience with JuiceFS, delivering a smoother, more convenient, and highly efficient workflow.

Here are the highlights of the new features in this release:

• juicefs quota: Set limits on the size and file count of directories to manage their capacity.
• juicefs clone: Quickly copy file metadata. This method saves time and data storage compared to a full copy, as it is a pure metadata operation.
• juicefs restore: Recover all deleted files within a specified time frame using a single command without the need for individual recovery operations.
• juicefs debug: Generate diagnostic reports with a single command to simplify troubleshooting.
• Directory statistics: View storage space and file count statistics in the tree structure.
• New metadata engine support: FoundationDB, a distributed key-value storage with support for distributed transactions.

This release involved 57 community contributors, who made a total of 726 commits. We extend our heartfelt appreciation to each contributor for their valuable contributions.

Below, we’ll provide a detailed overview of the new features and changes in this version.

Feature #1: Directory quotas

Quotas can be used to limit the maximum available storage space in the file system, preventing individual users from consuming excessive resources that may impact the overall system stability.

In earlier versions, JuiceFS only supported file system-level quotas. It was challenging for administrators to effectively control the resource usage of each user when the file system was shared among multiple users.

Therefore, in this version, we introduce the functionality of directory quotas. Specifically, administrators can set a quota threshold (hard limit) for any directory based on their requirements. Once the usage of the directory reaches or exceeds this threshold, any attempts to create or expand files within the directory will fail until the user deletes existing files or the administrator increases the quota threshold. Additionally, setting quotas for directories enables JuiceFS to track and record their usage, providing quick access to usage statistics for all files within the directory and its subdirectories when needed.

You can run juicefs quota to manage directory quotas. This command shares the same parameters as the existing file system quotas. The capacity can be limited using --capacity <val>, and the number of files can be restricted using --inodes. For example:

$ juicefs quota set $METAURL --path /test --capacity 1
+-------+---------+---------+------+-----------+-------+-------+
|  Path |   Size  |   Used  | Use% |   Inodes  | IUsed | IUse% |
+-------+---------+---------+------+-----------+-------+-------+
| /test | 1.0 GiB | 1.6 MiB |   0% | unlimited |   314 |       |
+-------+---------+---------+------+-----------+-------+-------+

The provided command sets a capacity quota of 1 GiB for the /test directory and simultaneously shows that the current usage in this directory is 1.6 MiB. Please be aware that when assigning a quota to an existing directory, there might be a delay due to the recursive calculation of the current usage. To inquire about the quota and current usage of a particular directory, you can use the quota get subcommand. For example:

$ juicefs quota get $METAURL --path /test
+-------+---------+---------+------+-----------+-------+-------+
|  Path |   Size  |   Used  | Use% |   Inodes  | IUsed | IUse% |
+-------+---------+---------+------+-----------+-------+-------+
| /test | 1.0 GiB | 1.6 MiB |   0% | unlimited |   314 |       |
+-------+---------+---------+------+-----------+-------+-------+

In addition, you can use the quota ls subcommand to retrieve a list of all the configured quotas. It’s important to note that the statistics for directory quotas are not updated in real time but instead have a slight delay. This delay is intentional to minimize the impact on system performance. Consequently, the following scenarios may occur: even if the directory usage has already reached the quota threshold, certain clients may still be able to write data within a short time frame (on the order of 10 seconds). Furthermore, if a client process terminates unexpectedly, the temporary usage information it recorded may not have been synchronized with the metadata engine, leading to inaccurate information.

To address this issue, JuiceFS offers the quota check subcommand, which allows you to inspect and rectify the calculated values in the quotas when necessary. Here is an example:

$ juicefs quota check $METAURL --path /test --repair
+-------+---------+---------+------+-----------+-------+-------+
|  Path |   Size  |   Used  | Use% |   Inodes  | IUsed | IUse% |
+-------+---------+---------+------+-----------+-------+-------+
| /test | 1.0 GiB | 3.2 MiB |   0% | unlimited |   317 |       |
+-------+---------+---------+------+-----------+-------+-------+

The restrictions on the number of files are similar to directory quotas, and we won't go into further detail. For detailes on how to use this feature, see Storage Quota.

Feature #2: Directory cloning

Sometimes, you may need to duplicate a set of files for various purposes. When dealing with a large number of files, using the cpcommand can be time-consuming and involve significant data copying within the object storage.

To address this issue, we introduce the directory cloning feature. It lets you quickly duplicate all files within a specified directory. The newly duplicated files have their own metadata but share the same data blocks as the original files, with the reference count incremented by one. Once the cloning process is complete, both sets of files are independent, allowing for separate modifications without mutual interference. Since cloning only involves metadata operations and doesn't require data copying, it’s significantly faster than a regular cp command. The command to perform the cloning operation is as follows:

$ juicefs clone /mnt/jfs/dir1 /mnt/jfs/dir2

Feature #3: One-command restore of Trash files

In JuiceFS, files in Trash are categorized based on their deletion time and assigned an index number corresponding to their original parent directory. This indexing system enables easy retrieval of files to their original locations when necessary.

However, in practical usage, reconstructing the directory structure using this information can be cumbersome and only suitable for manually restoring a small number of files. To address this issue, we introduce the juicefs restore command in this version, which helps organize these files automatically. Here's an example:

$ juicefs restore redis://localhost/1 2023-05-10-01 --put-back

The above command restores all files in the .trash/2023-05-10-01 directory to their original locations, based on the directory structure at the time of deletion. If the original parent directory doesn't exist or there are conflicting file names, a warning log will be printed, and those files will be skipped. You can manually restore them to the desired locations later.

Feature #4: One-command diagnostics collection

When JuiceFS encounters issues during runtime, new users often struggle to analyze the cause of the problem. To address this, we introduce the juicefs debug command in this version, which allows you to collect critical on-site information with a single command. This information includes host environment details, software versions, and runtime process status. For example:

$ juicefs debug /mnt/jfs --out-dir /tmp/jfs-debug

When the command completes and exits, you can find a .zip file named after the mount point and timestamp in the /tmp/jfs-debug directory. This file contains the diagnostic information collected during the process.

Feature #5: Quick usage statistics overview

In production environments, administrators need to regularly check the usage statistics of the file system or identify the directories that occupy the most space. Previously, in JuiceFS, this required administrators to manually calculate the usage of multiple directories (for example, using the du command) and then sort and filter the results. This approach was both cumbersome and time-consuming.

To address this issue, we introduce the juicefs summary command in this version. It lets you quickly view usage statistics for specified directories. For example:

$ juicefs summary /mnt/jfs --depth 1 --entries 5
+------+---------+------+-------+
| PATH |   SIZE  | DIRS | FILES |
+------+---------+------+-------+
| /    | 176 MiB |    9 |    20 |
| d2/  |  43 MiB |    1 |     5 |
| d4/  |  40 MiB |    3 |     4 |
| d5/  |  40 MiB |    1 |     4 |
| d3/  |  23 MiB |    1 |     4 |
| d1/  |  20 MiB |    1 |     2 |
| ...  |  10 MiB |    1 |     1 |
+------+---------+------+-------+

The command above calculates the usage of all top-level directories under /mnt/jfs and displays the top 5 entries sorted in descending order based on their size.

Other features

In this version, JuiceFS introduces several features to enhance system security and usability, including:

• Mapping the root user to a non-privileged user using the --root-squash option during mounting, reducing security risks and preventing accidental operations.
• Enabling partial support for ioctl during mounting using the --enable-ioctl option. This feature allows setting special flags to control file behavior, such as append-only (a) and immutable (i).
• Expanded support in the juicefs sync tool to synchronize object storage with JuiceFS files using the jfs:// prefix, without the need for mounting.

New metadata engine support: FoundationDB

In this release, we introduce a new metadata engine called FoundationDB. It’s a distributed database open-sourced by Apple, designed for efficient storage and management of large-scale structured data across multiple cluster servers. It offers high performance, scalability, and fault tolerance capabilities.

You can use FoundationDB as the metadata engine for JuiceFS by setting the Meta-URL to: fdb://<cluster_file_path>?prefix=<prefix>.

Here, cluster_file_path refers to the configuration file path of FoundationDB for server connection, and prefix is a user-defined string (similar to using TiKV) used to differentiate metadata spaces when multiple file systems or applications share the same FoundationDB cluster. For example:

$ juicefs format \
    --storage s3 \
    ... \
    "fdb:///etc/foundationdb/fdb.cluster?prefix=jfs" \
    pics

For details, see How to Set Up Metadata Engine.

Try us out!

To explore the full list of new features and enhancements, we encourage you to check out the release notes. If you're interested, you can download JuiceFS 1.1 Beta and give it a try. We aim to provide you with a smoother, more convenient, and highly efficient experience with JuiceFS.

If you have any questions or would like to share your thoughts, feel free to join our discussions on GitHub and community on Slack. We greatly appreciate the invaluable help, feedback, and support we receive from our users and community.