One side effect of SATA NCQ and SCSI TCQ is that some commands may be kept in the queue for an extended period of time without being serviced. The disk is optimizing for the closes accesses and if there is a single request at one side of the disk where all the other requests keep coming in at the other side that lonesome request will not be serviced for an extended period of time. The disks have a time limit that can be set and even enabled or disabled. Unfortunately this is a non-standard definition and each disk may do this differently. Hitachi have it in mode page 00 named “Command Aging”. The Ultrastar 7K4000 SAS spec documents the default value to be 2.4 seconds.
This means that for a naive user of the disk some requests may take upwards of 2 seconds and the process which depends on these requests will be starved. An additional distinction is that once we sent a command to the disk it is completely out of control for the OS and it can’t be pulled out and another placed instead.
A simple and yet effective technique to deal with this is documented in a thesis “Native command queuing and latency]thesis” by Lars Sondergaard Pedersen. In it he shows that by simply using a deadline on the requests and not placing new requests once a deadline for some request has expired we can force the disk to honor a lower request timeout, even if we do not know or even cannot change the Command Aging timeout of the disk.
This obviously affects max throughput but you may often need to let multiple different proccesses get their fair share of the disk IO and this may be a viable way to do it.
The deadline scheduler seems to do something relevant to this action but while it is mentioned in the thesis there is no discussion on why the deadline scheduler itself is insufficient to solve the problem.