HD Latency tool

Overview

The HD Latency tool tests the storage subsystem on a server by performing operations similar to what a MySQL/MariaDB database server would do. Sysbench can do this too, but hdlatency is a tiny standalone tool and therefore not hindered by dependencies. This is important as we work on client machines and don’t want to (and sometimes can’t) add lots of extra stuff.
InnoDB transaction logs and the MySQL/MariaDB binary log are sequential, and are fsynced after a write. The actual write performance is therefore restricted to the number of complete write/fsync operations a system can perform.

The InnoDB tablespace is accessed in a random access manner, optionally via direct I/O which bypasses the OS filesystem caching. That’s fine, as InnoDB has its own caching for this anyway (the InnoDB bufferpool). While generally direct I/O is faster, some devices/configurations show a degradation of performance in that mode, so we want to figure out which is more appropriate for a specific system.

Because InnoDB has its own cache, a storage subsystem’s cache becomes fairly irrelevant. If the data is in there, it’s likely to also be in the InnoDB cache. So typically, when InnoDB issues a read request, the data won’t be in the cache of the storage device. So a SAN, just like any other disk, will need to seek – and that’s a slow operation. In a nutshell, while a SAN may still have management and some other advantages, “speed” will almost certainly not be a factor. But worse than that, in many configurations a SAN has been found to perform worse than desktop SATA setups. As we know from other setups, that does not have to be the case.

Thus, we’ve found that the outcomes of these tests tend to cause controversy particularly with operators of SANs in datacenters, so much so we now often refer to hdlatency as “the tool that makes datacenter engineers cry”. The small footprint of hdlatency (just a few pages of C code) is easy to pass on so that sceptics can read it and try to poke holes in the methodology. (so far, no such holes have been identified.)

Usage

Usage: hdlatency [--quick] <testlabel> <filename> <MBfilesize> <seconds>

There is some output to stderr, and some to stdout which is in CSV format for easy import into a spreadsheet, and potential graphing. The test label field can help as an identifier.

We recommend specifying at least 1024 megabytes (1GB) for the file size, but possibly larger – you want it to be bigger than the cache of the storage device.

60 seconds can be enough, however particularly for actually stressing the cache a longer run is necessary.

Without the –quick option, hdlatency will try different blocksizes. With –quick, it’ll simply go for 16K blocks which is the default for InnoDB (although in XtraDB this can be configured, that’s generally only relevant for SSD and other special storage devices).

Download

The code is plain C with no dependencies, and is GPLv3 licensed.

  • hdlatency-2011-06-16.tar.gz (14KB) is a tarball with the source code, and 32 and 64 bit Linux binaries.
    • Please verify the integrity of your download with md5sum: 1ef40fd65b0b0b0c9c1abdaf4f5d4158
  • To grab the source from the repository, go to the hdlatency project on Launchpad.

References