JuiceFS 1.2: Gateway Upgrade, Enhanced Multi-User Permission Management

JuiceFS v1.2-beta1 is released, featuring significant improvements in user experience, bug fixes, and three new features:

• Enhanced gateway capabilities: Introduces Identity and Access Management (IAM) and event notifications to provide users with more secure, flexible, and automated data management and monitoring capabilities, suitable for multi-user environments and complex application scenarios.

• Support for POSIX access control lists (ACLs): Enables more powerful user permission management using Linux's built-in ACL tools (setfacl/getfacl).

• Support for smooth upgrades: Allows seamless upgrades without disrupting operations by remounting JuiceFS on the same mount point. This also facilitates online adjustment of mount parameters.

This release saw contributions from 41 contributors with 354 code merges. We appreciate the efforts of each contributor! In this post, we’ll introduce JuiceFS Gateway’s implementation and new features. In upcoming blog posts, we’ll dive into other features’ implementation principles and applications. Stay tuned.

About JuiceFS Gateway

JuiceFS stores files in chunks in the underlying object storage system. It provides you with POSIX interface access to files in JuiceFS. When you need to access files using the S3-compatible API, JuiceFS Gateway comes into play.

In JuiceFS v1.2-beta1, we’ve added two highly anticipated modules to the gateway:

• IAM: Supports multi-user management and access control as well as anonymous access control.
• Event notifications: Allows monitoring events happening on objects within buckets using bucket event notifications.

Implementation

The core service of JuiceFS Gateway is to expose the POSIX file system via the S3 API. Before v1.2, we integrated the MinIO Gateway module into our code. This module was based on the Apache license. This integration made JuiceFS operations similar to MinIO's native command minio gateway NAS. For users, JuiceFS could be seen as an implementation of MinIO Gateway. MinIO's other backend implementations include NAS and Hadoop.

In version v1.2, we’ve directly implemented the MinIO object interface and used JuiceFS as its backend storage. This approach provides a use experience close to native MinIO, inheriting many advanced features of MinIO. In this setup, JuiceFS acts as a local disk for MinIO server commands, similar to minio server /data1 in principle.

You can manage JuiceFS Gateway directly using MinIO's mc tool. Since MinIO is currently released under the Affero General Public License (AGPL) license, while JuiceFS uses its Apache license version, we recommend using the corresponding Apache licensed version of the mc tool to avoid potential compatibility issues. You can download it here.

The usage of the newly added modules is similar to the corresponding modules in MinIO. For details, see MinIO's document or the JuiceFS document. Next, we’ll briefly introduce these new features and their usage.

New features

IAM

User management

juicefs gateway allows adding new users using mc admin user add. You can use mc admin user to manage added users. It supports operations such as add, disable, enable, delete, list all users, display user information, and view user policies.

Service accounts

We allow adding service accounts for users. Each service account is associated with the user identity and inherits policies attached to its parent user or parent user's group. Each access key supports optional inline policies that can further restrict access to operations and resource subsets available to the parent user. You can use the mc admin user svcacct command to manage service accounts.

AssumeRole security token service

AssumeRole returns a set of temporary security credentials that the user can use to access gateway resources. AssumeRole requires the authorization credentials of an existing gateway user and returns temporary security credentials including access keys, secret keys, and security tokens. Applications can use these temporary security credentials to make signed calls to gateway API operations. The policy applied to these temporary credentials is inherited from the gateway user credentials. For usage methods, see AssumeRole security token service.

Permission management

Newly created users have no permissions by default and require assignment using mc admin policy to grant access. This command supports adding, deleting, updating, and viewing permissions as well as adding, deleting, and updating permissions for users. Additionally, the gateway has four built-in common policies:

• readonly: Read-only users.
• readwrite: Read-write users.
• writeonly: Write-only users.
• consoleAdmin: Read-write-admin users, where "admin" means the ability to use management APIs such as creating users.

You can use mc admin policy add to add set custom policies.

User group management

JuiceFS Gateway supports creating user groups, similar to Linux user groups, and uses mc admin group for management. You can set one or more users to a group and grant permissions uniformly to the group.

In addition to user-specific permissions, anonymous access management is also possible. This allows specific objects or buckets to be accessible to anyone. You can use the mc policy command to manage this functionality.

By default, the gateway has built-in support for four types of anonymous permissions:

• none: Disallows anonymous access (typically used to clear existing permissions).
• download: Allows anyone to read.
• upload: Allows anyone to write.
• public: Allows anyone to read and write.

Bucket event notifications

If you want to trigger certain actions based on events occurring on a bucket, you need the bucket event notification feature. You can use this feature to monitor events happening on objects within a storage bucket.

Currently supported object event types include:

• s3:ObjectCreated:Put
• s3:ObjectCreated:CompleteMultipartUpload
• s3:ObjectAccessed:Head
• s3:ObjectCreated:Post
• s3:ObjectRemoved:Delete
• s3:ObjectCreated:Copy
• s3:ObjectAccessed:Get

Supported global events include:

• s3:BucketCreated
• s3:BucketRemoved

You can use the mc client tool to set up and listen for event notifications through the event subcommand. The notification message sent by MinIO for publishing events is in JSON format. See the JSON structure.

Currently, storage bucket events can be published to the following destinations:

• Redis
• MySQL
• PostgreSQL
• Webhooks

For details, see Bucket event notifications.

Other features

This release has also introduced these gateway features:

• Object tagging support: Allows setting custom tags for objects. You can use mc tag to manage it.
• Background running: Supports running the gateway in the background. You can add the -d parameter during startup to enable this feature.

Give it a try!

Feel free to download and try JuiceFS 1.2-beta1 here. If you have any questions, join JuiceFS discussions on GitHub and our community on Slack.