Metrics
JuiceFS provided a Prometheus API for each file system, this page only contains some of the mostly used metrics. For the full metrics list, visit JuiceFS console, navigate to the "Monitor" page, and click "Prometheus API".
tip
Some of the metrics come with multiple subnodes, and are omitted in this section. e.g. juicefs_fuse_ops_lookup、juicefs_fuse_ops_open are all included in juicefs_fuse_ops_<name>.
On the other hand, metrics introduced in this chapter all start with the juicefs_ prefix, this is different in on-prem Grafana data source, for example:
juicefs_trash_sizeis calledjfs_stat_trash_sizejuicefs_operationDurationis calledmount_operationDuration
So when you encounter trouble searching for the desired metrics, strip the prefix or suffix and try again.
File system
Labels
| Name | Description |
|---|---|
name | file system name |
Metrics
| Name | Description | Unit |
|---|---|---|
juicefs_size | size of files | byte |
juicefs_inodes | number of inodes | |
juicefs_trash_size | size of files in Trash | byte |
juicefs_trash_files | number of files in Trash |
Client
Labels
| Name | Description |
|---|---|
name | file system name |
host | client hostname |
ip | client IP address |
mountpoint | path of mount point |
Metrics
Operating system
| Name | Description | Unit |
|---|---|---|
juicefs_uptime | uptime | second |
juicefs_cpuusage | overall CPU usage time | microsecond |
juicefs_memusage | current RSS memory usage | byte |
juicefs_heapSys | Total memory allocated to the client process, same as Sys | byte |
juicefs_heapInuse | Same as HeapInuse | byte |
juicefs_handles | Number of file handlers (descriptors) | |
juicefs_threads | Number of threads in the client process, see ThreadCreateProfile | |
juicefs_goroutines | Number of goroutines in the client process | |
juicefs_gcPause | Same as PauseTotalNs | nanosecond |
Metadata service
| Name | Description | Unit |
|---|---|---|
juicefs_metaDuration | Duration of Metadata requests | microsecond |
juicefs_metaRequest | Number of Metadata requests | |
juicefs_meta | Sum of all meta metrics, not very useful | |
juicefs_meta_operations | Number of metadata operations | |
juicefs_meta_bytes_sent | Metadata requests sent traffic | byte |
juicefs_meta_bytes_received | Metadata requests received traffic | byte |
juicefs_meta_packets_sent | Metadata requests sent packets | |
juicefs_meta_packets_received | Metadata requests received packets | |
juicefs_meta_reconnects | Number of metadata service reconnects | |
juicefs_meta_usec_ping | Ping latency between client and metadata service | microsecond |
File access
| Name | Description | Unit |
|---|---|---|
juicefs_read_bytes | Total read bytes, different from juicefs_get_bytes, this is counted in the file system level | byte |
juicefs_write_bytes | Total write bytes, different from juicefs_put_bytes, this is counted in the file system level | byte |
juicefs_operations | Number of file operations | |
juicefs_operationDuration | File operation latency | microsecond |
juicefs_operationErrors | Number of file operation errors, use this metric to monitor application I/O errors | |
juicefs_operationSlows | Number of slow operations (above 10s) | |
juicefs_operationInterrupted | Number of interruptted operations | |
juicefs_fuse_ops | Number of file operations, roughly the same as juicefs_operations, but counted by summing up all different operation types | |
juicefs_fuse_ops_<name> | Number of single file operations, e.g. getattr、lookup、open | |
juicefs_openfiles | Number of opened files |
Buffer
| Name | Description | Unit |
|---|---|---|
juicefs_totalBufferUsed | Used buffer size | byte |
juicefs_readBufferUsed | Used read buffer size | byte |
juicefs_readahead_bytes | The total amount of readahead data written from the buffer to the kernel page cache | byte |
Metadata cache
| Name | Description | Unit |
|---|---|---|
juicefs_meta_dircache | The total number of all operations that hit client metadata cache | |
juicefs_meta_dircache_<name> | The total number of various operations that hit the client metadata cache, such as getattr, open, lookup | |
juicefs_meta_dircache<zone-number>_dirs | The number of directories for client metadata cache with <zone-number> (such as 0, 1) | |
juicefs_meta_dircache<zone-number>_inodes | The number of inodes for client metadata cache with <zone-number> (such as 0, 1) | |
juicefs_symlink_cache | The number of all symbolic link cache operations | |
juicefs_symlink_cache_<name> | The number of specific symbolic link cache operation, such as inserts, search_hits, search_misses |
Local cache
tip
It is recommended to read the "Cache" document first to help understand the specific meaning of the following metrics.
| Name | Description | Unit |
|---|---|---|
juicefs_blockcache_blocks | Number of cache blocks | |
juicefs_blockcache_bytes | Size of cache blocks | byte |
juicefs_blockcache_pri0 | Total size of cache blocks where priority is 0 | byte |
juicefs_blockcache_pri1 | Total size of cache blocks where priority is 1 | byte |
juicefs_blockcache_pri2 | Total size of cache blocks where priority is 2 | byte |
juicefs_blockcache_pri3 | Total size of cache blocks where priority is 3 | byte |
juicefs_blockcache_hits | Number of cache block hits | |
juicefs_blockcache_hitBytes | Bytes of cache block hits | byte |
juicefs_blockcache_miss | Number of missed cache block hits | |
juicefs_blockcache_missBytes | Size of missed cache block hits | byte |
juicefs_blockcache_evict | Number of cache block evictions | |
juicefs_blockcache_evictBytes | Size of cache block evictions | byte |
juicefs_blockcache_evictDur | Duration of cache block evictions | microsecond |
juicefs_blockcache_readDuration | Latency in reading cache blocks from cache disk | microsecond |
juicefs_blockcache_write | The total number of cache blocks written to the cache disk (including read cache blocks and write cache blocks) | |
juicefs_blockcache_writeBytes | The total size of cache blocks written to the cache disk (including read cache blocks and write cache blocks) | byte |
juicefs_blockcache_writeDuration | Latency in writing cache blocks (including read cache blocks and write cache blocks) to the cache disk | microsecond |
juicefs_blockcache_stageBlocks | The total number of write cache blocks waiting to be uploaded | |
juicefs_blockcache_stageBlockBytes | Total size of write cache blocks waiting to be uploaded | byte |
juicefs_blockcache_stageBlockDelay | The waiting time for the cache block from writing to the cache disk to the beginning of uploading to object storage | microsecond |
Distributed cache
In the cache group, there will be corresponding metrics for "sender" and "receiver" for reading data and writing data, also called "server" and "client" in later descriptions, note that the phrase "server" in this context refers to the party serving the cache blocks in the cache group, and the phrase "client" stands for the party receiving the cache blocks. In a cache group, servers and clients can actually be the same set of JuiceFS clients, but if the cache group is deployed as a dedicated cache cluster, then in this architecture, clients do not share their cache, hence servers and clients are two separate sets of JuiceFS clients.
For example, when reading files through a cache group, member nodes will not only receive data from other members, but also share and send their own cache data to other members.
There are two sets of mutually contrasting metrics in distributed cache metrics, corresponding to different application scenarios:
remotecache_getandremotecache_send: correspond to the scenario where the client reads data from the distributed cacheremotecache_receiveandremotecache_put: correspond to the scenario where the client writes data to the distributed cache (the--fill-group-cacheoption needs to be enabled), or the scenario of balancing data in the distributed cache.
Therefore, under normal circumstances, if you sum up metrics of all cache group members (including --no-sharing clients), the size of remotecache_sendBytes should be roughly equal to remotecache_getBytes. If it is found that remotecache_sendBytes is much larger that remotecache_getBytes, it means that there are errors during data transfer, check remotecache_errors metric and client logs for causes.
| Name | Description | Unit |
|---|---|---|
juicefs_remotecache_errors | Number of all failed distributed cache requests | |
juicefs_remotecache_get | Number of requests for cache group client to read data | |
juicefs_remotecache_getBytes | Size of cache group client read request | byte |
juicefs_remotecache_getDuration | Latency of cache group client read request | microsecond |
juicefs_remotecache_send | Number of requests for data sent by the cache group server | |
juicefs_remotecache_sendBytes | Size of the data sent by the cache group server | byte |
juicefs_remotecache_sendDuration | Latency of the cache group server send requests, including the latency in reading the local cache (juicefs_blockcache_readDuration). If there is no hit in the local cache, it also includes the request time of the object storage (juicefs_objectDuration_get). If the object storage response is slow or there is a speed limit, this metric will increase. | microsecond |
juicefs_remotecache_put | Number of requests for cache group client to write data | |
juicefs_remotecache_putBytes | Size of cache group client write request | byte |
juicefs_remotecache_putDuration | Latency of cache group client write request | microsecond |
juicefs_remotecache_receive | Number of received write requests for the cache group server | |
juicefs_remotecache_receiveBytes | Size of the data received by the cache group server | byte |
juicefs_remotecache_recvDuration | Latency of the write requests received for cache group server | microsecond |
juicefs_remotecache_delete | Number of requests for cache group client to delete data | |
juicefs_remotecache_warmup | Number of requests for cache group client warmup data | |
juicefs_remotecache_redirect Added in v5.0.17 | The number of times the cache group server redirects data | |
juicefs_remotecache_redirectBytes Added in v5.0.17 | Size of cache group server redirect data | byte |
juicefs_blockcache_transfer | The number of data blocks sent to other cache nodes during data rebalancing | |
juicefs_blockcache_transferBytes | The total size of data blocks sent to other cache nodes during data rebalancing | byte |
juicefs_blockcache_transferDur | Latency in sending data blocks to other cache nodes during data rebalancing | microsecond |
juicefs_blockcache_transferScanDur | Latency in scanning all data block keys during data rebalancing | microsecond |
Object storage
| Name | Description | Unit |
|---|---|---|
juicefs_object | Number of total object storage requests | |
juicefs_object_get | Number of GetObject requests | |
juicefs_object_put | Number of PutObject requests | |
juicefs_object_delete | Number of DeleteObject requests | |
juicefs_object_copy | Number of CopyObject requests | |
juicefs_object_head | Number of HeadObject requests | |
juicefs_object_list | Number of ListObjects requests | |
juicefs_object_error | Number of object storage requests failures | |
juicefs_objectDuration | Duration of all object storage requests | microsecond |
juicefs_objectDuration_get | Duration of GetObject requests | microsecond |
juicefs_objectDuration_getDelay | GetObject requests wait, wait is caused by client token speed control or congestion caused by low --max-downloads | microsecond |
juicefs_objectDuration_put | Duration of PutObject requests | microsecond |
juicefs_objectDuration_putDelay | PutObject requests wait, wait is caused by client token speed control or congestion caused by low --max-uploads | microsecond |
juicefs_objectDuration_delete | Duration of DeleteObject requests | microsecond |
juicefs_objectDuration_copy | Duration of CopyObject requests | microsecond |
juicefs_objectDuration_head | Duration of HeadObject requests | microsecond |
juicefs_objectDuration_list | Duration of ListObjects requests | microsecond |
juicefs_get_bytes | Size of GET requests | byte |
juicefs_put_bytes | Size of PUT requests | byte |
juicefs_currentReq_get | Number of GetObject requests in progress | |
juicefs_currentReq_put | Number of PutObject requests in progress (excluding requests sent by background jobs) | |
juicefs_currentReq_delete | Number of DeleteObject requests in progress |
Shared block device
tip
It is recommended to read the "Shared Block Device" document first to help understand the specific meaning of the following metrics.
| Name | Description | Unit |
|---|---|---|
juicefs_block_read | The number of times data was read from the block device | |
juicefs_block_readByteDur | The latency in reading data from block device | microsecond |
juicefs_block_readBytes | The total amount of data read from the block device | byte |
juicefs_block_write | The number of times data is written to the block device | |
juicefs_block_writeByteDur | The latency in writing data to block device | microsecond |
juicefs_block_writeBytes | The total amount of data written to the block device | byte |
Background job
tip
It is recommended to read "Background job", "Convert" and "Shared Block Device" document to help understand the specific meaning of the following metrics.
| Name | Description | Unit |
|---|---|---|
juicefs_bgjobs | The number of all background jobs being executed | |
juicefs_bgjobs_compact | The number of compaction jobs being executed | |
juicefs_bgjobs_convert | The number of convert jobs being executed | |
juicefs_bgjobs_delete | The number of deletion jobs being executed | |
juicefs_bgjobs_migrate | The number of jobs being executed to migrate data from block device to object storage | |
juicefs_bgjobs_sync | The number of data replication jobs being performed by the mirror cluster client | |
juicefs_compacts | The number of execution times of the compaction job | |
juicefs_compact_bytes | The total size of data blocks processed by the compaction job | byte |
juicefs_compact_duration | The execution time of compaction job | microsecond |
juicefs_migrates | The number of executions of jobs to migrate data from block device to object storage | |
juicefs_migrate_bytes | The total size of data blocks processed by the job of migrating data from block device to object storage | byte |
juicefs_migrate_duration | The execution time of job to migrate data from block device to object storage | microsecond |
juicefs_syncs | The number of execution times of the data replication job by the mirror cluster client | |
juicefs_sync_bytes | The total size of data blocks processed by the data replication job of the mirror cluster client | byte |
juicefs_sync_duration | The execution time of data replication jobs on the mirror cluster client | microsecond |
Data synchronization
| Name | Description | Unit |
|---|---|---|
juicefs_sync_scanned | Number of all objects scanned from the source | |
juicefs_sync_handled | Number of objects from the source that have been processed | |
juicefs_sync_pending | Number of objects waiting to be synchronized | |
juicefs_sync_copied | Number of objects that have been synchronized | |
juicefs_sync_copied_bytes | Total size of data that has been synchronized | byte |
juicefs_sync_skipped | Number of objects that skipped during synchronization | |
juicefs_sync_failed | Number of objects that failed during synchronization | |
juicefs_sync_deleted | Number of objects that deleted during synchronization | |
juicefs_sync_checked | Number of objects that have been verified by checksum during synchronization | |
juicefs_sync_checked_bytes | Total size of data that has been verified by checksum during synchronization | byte |
