StorageServerTuning

Source!

Tips and Recommendations for Storage Server Tuning

Here are some tips and recommendations on how to improve the performance of your storage servers. As usual, the optimal settings depend on your particular hardware and usage scenarios, so you should use these settings only as a starting point for your tuning efforts.

Note: Some of the settings suggested here are non-persistent and will be reverted after the next reboot. To keep them permanently, you could add the corresponding commands to /etc/rc.local, use /etc/sysctl.conf or create udev rules to reapply them automatically when the machine boots.

Partition Alignment & RAID Settings of Local File System

To get the maximum performance out of your RAID arrays and SSDs, it is important to set the partition offset according to the native alignment. See here for a walk-through about partition alignment and creation of a RAID-optimized local file system: Partition Alignment Guide

A very simple and commonly used method to achieve alignment without the challenges of partition alignment is to completely avoid partitioning and instead create the file system directly on the device, e.g.:

 

$ mkfs.xfs /dev/sdX

Storage Server Throughput Tuning

In general, BeeGFS can be used with any of the standard Linux file systems.

Using XFS for your storage server data partition is generally recommended, because it scales very well for RAID arrays and typically delivers a higher sustained write throughput on fast storage, compared to alternative file systems. (There also have been significant improvements of ext4 streaming performance in recent Linux kernel versions).

However, the default Linux kernel settings are rather optimized for single disk scenarios with low IO concurrency, so there are various settings that need to be tuned to get the maximum performance out of your storage servers.

Formatting Options

Make sure to enable RAID optimizations of the underlying file system, as described in the last section here: Create RAID-optimized File System

While BeeGFS uses dedicated metadata servers to manage global metadata, the metadata performance of the underlying file system on storage servers still matters for operations like file creates, deletes, small reads/writes, etc. Recent versions of XFS (similar work in progress for ext4) allow inlining of data into inodes to avoid the need for additional blocks and the corresponding expensive extra disk seeks for directories. In order to use this efficiently, the inode size should be increased to 512 bytes or larger.

Example: mkfs for XFS with larger inodes on 8 disks (where the number 8 does not include the number of RAID-5 or RAID-6 parity disks) and 128KB chunk size:

 

$ mkfs.xfs -d su=128k,sw=8 -l version=2,su=128k -isize=512 /dev/sdX

Mount Options

Enabling last file access time is inefficient, because it means that the file system needs to update the timestamp by writing data to the disk even though the user only actually read file contents or even when the file contents were already cached in memory and actually no disk access would have been necessary at all. (Note: Recent Linux kernels have switched to a new "relative atime" mode, so setting noatime might not be necessary in these cases.)
If your users don't need last access times, you should disable them by adding "noatime" to your mount options.

Increasing the number of log buffers and their size by adding logbufs and logbsize mount options allows XFS to generally handle and enqueue pending file and directory operations more efficiently.

There are also several mount options for XFS that are intended to further optimize streaming performance on RAID storage, such as largeio, inode64, and swalloc.

If you are using XFS and want to go for optimal streaming write throughput, you might also want to add the mount option allocsize=131072k to reduce the risk of fragmentation for large files.

If your RAID controller has a battery-backup-unit (BBU), adding the mount option nobarrier for XFS or ext4 can significantly increase throughput.

Example: Typical XFS mount options for an BeeGFS storage server with a RAID controller battery:

 

$ mount -onoatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier /dev/sdX <mountpoint>

 

 

IO Scheduler

First, set an appropriate IO scheduler for file servers:

$ echo deadline > /sys/block/sdX/queue/scheduler

Now give the IO scheduler more flexibility by increasing the number of schedulable requests:

$ echo 4096 > /sys/block/sdX/queue/nr_requests

To improve throughput for sequential reads, increase the maximum amount of read-ahead data. The actual amount of read-ahead is adaptive, so using a high value here won't harm performance for small random access.

$ echo 4096 > /sys/block/sdX/queue/read_ahead_kb

Virtual memory settings

To avoid long IO stalls (latencies) for write cache flushing in a production environment with very different workloads, you will typically want to limit the kernel dirty (write) cache size:

 

$ echo 5 > /proc/sys/vm/dirty_background_ratio
$ echo 10 > /proc/sys/vm/dirty_ratio

Only for special use-cases: If you are going for optimial sustained streaming performance, you may instead want to use different settings that start asynchronous writes of data very early and allow the major part of the RAM to be used for write caching. (For generic use-cases, use the settings described above, instead.)

 

$ echo 1 > /proc/sys/vm/dirty_background_ratio
$ echo 75 > /proc/sys/vm/dirty_ratio

Assigning slightly higher priority to inode caching helps to avoid disk seeks for inode loading:

 

$ echo 50 > /proc/sys/vm/vfs_cache_pressure

Buffering of file system data requires frequent memory allocation. Raising the amount of reserved kernel memory will enable faster and more reliable memory allocation in critical situations. Raise the corresponding value to 64MB if you have less than 8GB of memory, otherwise raise it to at least 256MB:

 

$ echo 262144 > /proc/sys/vm/min_free_kbytes

 

 Transparent huge pages can cause performance degradation under high load, due to the frequent change of file system cache memory areas. For RedHat 6.x and derivatives, it is recommended to disable transparent huge pages support, unless huge pages are explicity requested by an application:

$ echo madvise > /sys/kernel/mm/redhat_transparent_hugepage/enabled
$ echo madvise > /sys/kernel/mm/redhat_transparent_hugepage/defrag

With recent mainline kernel versions:

$ echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
$ echo madvise > /sys/kernel/mm/transparent_hugepage/defrag

 

Controller Settings

Optimal performance for hardware RAID systems often depends on large IOs being sent to the device in a single large operation. Please refer to your hardware storage vendor for the corresponding optimal size of /sys/block/sdX/max_sectors_kb. It is typically good if this size can be increased to at least match your RAID stripe set size (i.e. chunk_size x number_of_disks):

 

$ echo 1024 > /sys/block/sdX/queue/max_sectors_kb

Furthermore, high values of sg_tablesize (/sys/class/scsi_host/*/sg_tablesize) are recommended to allow large IOs. Those values depend on controller firmware versions, kernel versions and driver settings.

System BIOS & Power Saving

To allow the Linux kernel to correctly detect the system properties and enable corresponding optimizations (e.g. for NUMA systems), it is very important to keep your system BIOS updated.

The dynamic CPU clock frequency scaling feature for power saving, which is often enabled by default, has a high impact on latency. Thus, it is recommended to turn off dynamic CPU frequency scaling. Ideally, this is done in the machine BIOS (see Intel SpeedStep or AMD PowerNow), but it can also be done at runtime, e.g. via:

 

$ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null