POSIX Compatibility Comparison Among Four File Systems on the cloud

2022-10-21

Zhijian

POSIX compatibility is an indispensable criterion when choosing file system. Recently, we conducted a test on POSIX compatibility among GCP Filestore, Amazon EFS, Azure Files and JuiceFS.

About POSIX

POSIX ( Portable Operating System Interface) is the most widely used interface standards for operating systems, including file systems. If you want to learn more about POSIX, please refer to the Quora question and answer "What does POSIX conformance/compliance mean in the distributed systems world?"

Test Method

One popular POSIX compatibility test suites is pjdfstest, derived from FreeBSD and also applicable to systems such as Linux.

Test Results

The test results are shown below, JuiceFS failed with 0 cases, showing the best compatibility. GCP Filestore is the second best, with two failures. Amazon EFS failed with several orders of magnitude larger test cases compared to other products.

Note that, for the sake of comparison, logarithmic coordinates are used for the horizontal coordinates of the result figure.

Failure Use Case Analysis

GCP Filestore

The GCP Filestore failed2 tests in total, one in each of the unlink and utimensat categories.

The first one is in the unlink test set unlink/14.t, and the corresponding log is as follows.

/root/pjdfstest/tests/unlink/14.t ...........
not ok 4 - tried 'open pjdfstest_b03f52249a0c653a3f382dfe1237caa1 O_RDONLY : unlink pjdfstest_b03f52249a0c653a3f382dfe1237caa1 : fstat 0 nlink', expected 0, got 1

This test set （unlink/14.t） is used to verify the behavior of a file when it is deleted in the open state.

desc="An open file will not be immediately freed by unlink"

The operation of deleting a file actually corresponds to unlink at the system level, which removes the link from filename to inode, and then minus the corresponding nlink by 1. This test is to verify this.

# A deleted file's link count should be 0
expect 0 open ${n0} O_RDONLY : unlink ${n0} : fstat 0 nlink

The contents of a file are only really deleted when the number of links (nlink) is reduced to 0 and there are no open file descriptors (fd) pointing to the file. If nlink is not updated correctly, it may result in files that should be deleted remaining on the system.

The other one is in the utimensat test set utimensat/09.t, which corresponds to the following log.

/root/pjdfstest/tests/utimensat/09.t ........
not ok 5 - tried 'lstat pjdfstest_909f188e5492d41a12208f02402f8df6 mtime', expected 4294967296, got 4294967295

This test case requires 64-bit timestamp support. GCP Filestore supports 64-bit timestamp, but it will be reduced by 1 on top of that. So it should not affect the use of this test case even though it fails.

Amazon EFS

Amazon Elastic File System (EFS) failed 21.49% of the pjdfstest tests, with failure use cases covering almost all categories.

Amazon EFS supports mounting via NFS, but the support for NFS features is not complete. For example, EFS does not support block and character devices, which directly led to the failure of a large number of test cases in pjdfstest. After excluding these two types of files, there are still hundreds of different categories of failure.

Azure Files

Azure Files has a failure rate of 55%, indicating that some basic POSIX scenarios may have incompatibility issues.

JuiceFS passed all test items, performed the best in terms of compatibility.
Google Filestore was the next, only failing in two categories, one of which did not affect actual usage.
Amazon EFS has the worst compatibility with Azure Files, with a large number of compatibility tests failing, including several test cases with serious security risks, so it is recommended to do a security assessment before use.