TiKV Best Practices

TiKV uses the Raft consensus algorithm to ensure data consistency across replicas and high availability. Therefore, deploy at least three replicas in production to ensure data safety and service stability.

TiKV scales horizontally and is suitable for large-scale file systems with demanding performance requirements.

Garbage collection

TiKV natively supports multi-version concurrency control (MVCC). When newly written data overwrites old data, the old data is retained alongside the new data instead of being replaced, and the versions are distinguished by timestamps. Garbage collection (GC) removes old data that is no longer needed.

JuiceFS configuration

TiKV uses a cluster variable named safe-point (a timestamp) to decide whether to remove old versions created before a given time. Before JuiceFS v1.0.4, JuiceFS did not set safe-point, so the TiKV metadata engine depended on TiDB for garbage collection. Starting with v1.0.4, JuiceFS clients periodically set safe-point. By default, versions older than three hours are removed. You can adjust this interval by setting the gc-interval parameter in the metadata URL when mounting.

• Mount log with the default gc-interval

  > sudo ./juicefs mount tikv://localhost:2379 ~/mnt/jfs
  2023/04/06 20:23:34.741432 juicefs[17286] <INFO>: Meta address: tikv://localhost:2379 [interface.go:491]
  2023/04/06 20:23:34.741561 juicefs[17286] <INFO>: TiKV gc interval is set to 3h0m0s [tkv_tikv.go:84]
  ...

• Mount log after setting gc-interval

  > sudo ./juicefs mount tikv://localhost:2379\?gc-interval=1h ~/mnt/jfs
  2023/04/06 20:25:58.134999 juicefs[17395] <INFO>: Meta address: tikv://localhost:2379?gc-interval=1h [interface.go:491]
  2023/04/06 20:25:58.135113 juicefs[17395] <INFO>: TiKV gc interval is set to 1h0m0s [tkv_tikv.go:84]
  ...

Set safe-point manually

In addition to the periodic updates performed by JuiceFS clients, you can use the gc subcommand to set safe-point manually.

> ./juicefs gc -v tikv://localhost:2379\?gc-interval=1h --delete
...
2023/04/06 20:41:57.145692 juicefs[18531] <DEBUG>: TiKV GC returns new safe point: 440606737600086016 (2023-04-06 19:41:57.139 +0800 CST) [tkv_tikv.go:248]
...

tip

This command also removes leaked objects and pending-deletion objects generated by JuiceFS. Refer to Status Check and Maintenance to determine whether you should use it.

TiKV garbage collection modes

• gc-worker

  You can enable gc-worker through the TiKV configuration. In gc-worker mode, garbage is collected promptly, but the large amount of additional disk I/O might affect metadata engine performance.

  [gc]
  enable-compaction-filter = false

• compaction-filter

  By default, TiKV uses compaction filter for garbage collection. GC is performed during RocksDB compaction instead of by a separate GC worker thread. This avoids the additional disk reads caused by GC and prevents large numbers of deletion markers left by removed old versions from degrading sequential scan performance.

  Because this mode relies on RocksDB compaction, garbage is not collected immediately after safe-point is set. Subsequent writes must trigger compaction before GC can occur. To trigger GC manually, use tikv-ctl to compact the cluster and thereby trigger global GC.

  > tikv-ctl --pd 127.0.0.1:2379 compact-cluster -b -c default,lock,write

Metadata backup

For large-scale file systems, increase tikv_gc_life_time. Otherwise, a backup might fail with the error GC life time is shorter than transaction duration.

Runtime environment and tuning

Hardware selection

According to TiDB software and hardware requirements, TiKV can be deployed and run on 64-bit general-purpose server hardware based on Intel x86-64 or ARM architectures. The following requirements and recommendations apply to server hardware in development, test, and production environments, excluding resources used by the operating system:

• Development and test environments

  Component	CPU	Memory	Local storage	Network	Minimum number of instances
  PD	4 cores+	8 GB+	SAS, 200 GB+	Gigabit NIC	1
  TiKV	8 cores+	32 GB+	SSD, 200 GB+	Gigabit NIC	3

  note

  • For performance testing, avoid low-performance storage and network hardware, which might distort the test results.
  • NVMe SSDs are recommended for TiKV to achieve faster reads and writes.

• Production environments

  Component	CPU	Memory	Local storage	Network	Minimum number of instances
  PD	8 cores+	16 GB+	SSD	10 Gigabit NIC (two preferred)	3
  TiKV	16 cores+	64 GB+	SSD	10 Gigabit NIC (two preferred)	3

  note

  For TiKV disks, use no more than 2 TB per PCIe SSD or 1.5 TB per regular SSD.

Network requirements

TiKV requires the following network ports. Administrators should open the appropriate ports on the network and hosts depending on the component deployment plan:

Component	Default port	Description
TiKV	20160	TiKV communication port
TiKV	20180	TiKV status reporting port
PD	2379	Communication port for TiDB and PD
PD	2380	Communication port between PD cluster nodes

Disk space requirements

Component	Disk space requirement	Healthy utilization
PD	Reserve at least 20 GB each for data and log disks	Below 90%
TiKV	Reserve at least 100 GB each for data and log disks	Below 80%

Hardware tuning

Database systems have specific hardware requirements, and components such as TiKV have minimum requirements for CPU, memory, disks, and NICs. With those requirements met, this section discusses hardware parameter tuning, mainly based on database hardware tuning.

CPU

• CPU selection

  Workloads can generally be categorized as compute-intensive or storage-intensive. Compute-intensive workloads usually need more CPU cores and higher clock speeds, while storage-intensive workloads can use slightly less powerful CPUs. In typical JuiceFS use cases, PD and TiKV are primarily storage-intensive and do not have heavy compute loads. Plan accordingly to make hardware procurement more cost-effective.

• CPU architecture: x86/ARM

  The x86 architecture is used by Intel and AMD CPUs. It uses a complex instruction set and is currently the most common server CPU architecture. ARM CPUs are used in mobile phones, Mac laptops, and servers from vendors. Most companies currently purchase x86-64 CPUs, while also validating ARM servers for web and database applications. TiKV supports both architectures, so choose based on your deployment requirements.

• NUMA CPU affinity

  On a multi-core CPU, cores are distributed across different NUMA nodes. Each NUMA node has its own local main memory, and accessing local memory is faster than accessing memory across NUMA nodes. Enabling NUMA allows the system to preferentially use nearby memory. This configuration is recommended when deploying multiple nodes on a single machine.

• Dynamic CPU frequency scaling

  cpufreq is a module that dynamically adjusts CPU frequency and supports five governors. To ensure service performance, use the performance governor, which keeps the CPU at its highest supported operating frequency for optimal performance. The default is usually powersave. You can change it with cpupower frequency-set.

Memory

• Disable swap

  Swap uses disk to handle memory access after a certain threshold is reached. It is controlled by vm.swappiness, whose default value is 60, meaning that swap begins to be used when 40% of system memory is in use. TiKV requires sufficient memory. If memory is insufficient, do not use swap as a buffer because it degrades performance. Disabling system swap is recommended.

• Set min_free_kbytes

  The min_free_kbytes kernel parameter controls how much memory should remain free instead of being used by the file system cache. Normally, the kernel uses almost all free memory for the file system cache and releases it as needed for processes. Because databases perform many allocations in shared memory, the default kernel value might cause unexpected out-of-memory (OOM) kills. On systems with more than 40 GB of memory, set this parameter to at least 1 GB but no more than 5% of total memory. This ensures that Linux always keeps enough memory available.

• Disable Transparent Huge Pages (THP)

  Database memory access patterns are often sparse rather than contiguous. When higher-order memory is heavily fragmented, allocating THP pages can incur high latency. Enabling direct memory compaction for THP can also cause a sharp increase in system CPU utilization. Therefore, disabling THP is recommended.

• Adjust the virtual memory dirty_ratio and dirty_background_ratio parameters

  dirty_ratio is the absolute percentage limit for dirty pages. When the total dirty page cache reaches this percentage of total system memory, the system starts using pdflush to write the dirty page cache to disk. The default is 20%. Reaching this limit might cause application processes to wait for I/O, but adjustment is usually unnecessary.

  dirty_background_ratio is the percentage at which the system starts writing dirty page cache to disk in the background. The default is 10%. If background flushing is slow while data is written quickly, the dirty_ratio limit can easily be reached. Adjustment is usually unnecessary. For high-performance SSDs such as NVMe devices, a lower value can improve memory reclamation efficiency.

Data storage

Disk selection

Common disk options include:

• SAS is generally used with RAID controllers to create RAID 0, 1, 10, or 5 arrays.
• SATA supports hot swapping and provides a maximum interface speed of 6 Gbit/s.
• PCIe provides a higher transfer rate of 8 Gbit/s and supports multiple lanes, allowing bandwidth to scale linearly. It has traditionally been used for NICs and graphics cards. The three interfaces above use different protocols: AHCI was designed for SAS and SATA, while NVMe was designed for PCIe SSDs and offers better performance. SSDs of this type are generally used for core, I/O-intensive databases.
• Persistent memory, such as Intel Optane, provides rich low-level interfaces but is expensive. Consider it for workloads that require the highest possible write performance.

I/O schedulers

noop (no operation)

noop is the simplest I/O scheduler in the kernel. It places I/O requests in a FIFO queue and processes them one by one, while merging some requests that access contiguous disk locations. This scheduler is particularly suitable for applications that do not want the scheduler to reorder I/O requests, because kernel I/O scheduling introduces performance overhead. High-speed I/O devices such as NVMe SSDs can submit requests directly to the hardware for better performance.

CFQ

Completely Fair Queuing (CFQ) attempts to provide fair I/O scheduling among the processes that issue I/O. It assigns each process a time slice during which the process can issue I/O requests. Moving the time slice among processes gives every process a fair opportunity to issue requests. However, if a small number of processes issue large volumes of intensive I/O requests, I/O performance degrades noticeably.

deadline

The deadline scheduler focuses on I/O request latency and assigns a deadline to every request. Read and write requests are placed in separate queues. Reads are processed first by default, unless a write request is approaching its deadline. When only a small number of processes issue I/O requests, deadline can provide higher I/O throughput than CFQ.

FAQ

How can I avoid continuous transaction retries when multiple machines concurrently read from and write to the same directory?

When multiple clients frequently create or delete subdirectories in the same directory, transactions might be retried continuously. Starting with JuiceFS v1.1, the --skip-dir-nlink value mount option specifies the number of retries before skipping the directory nlink check. The default is 20. Reduce this value appropriately, or set it to 0 to disable retries and prevent continuous transaction retries. For details, see metadata mount options.