nr_requests device queue depth too low
These parameters are too low. Normally on high performance hardware values of about 1,000 or more are more reasonable.
Looking at some graphs it looks like dm-0 is going up to about 200-300 pending requests when the system is moderately busy. Our experiences with the bottlenecking when our utilization hits 95% is probably due to this.
/sys/block/dm-0/queue/nr_requests : 128
/sys/block/sda/queue/nr_requests : 128
/sys/block/sdb/queue/nr_requests : 128
/sys/block/sdc/queue/nr_requests : 128
/sys/block/sdd/queue/nr_requests : 128
/sys/block/sde/queue/nr_requests : 128
/sys/block/sdf/queue/nr_requests : 128
/sys/block/sdg/queue/nr_requests : 128
/sys/block/sdh/queue/nr_requests : 128
/sys/block/sdi/queue/nr_requests : 128
/sys/block/sdj/queue/nr_requests : 128
/sys/block/sdk/queue/nr_requests : 128
/sys/block/sdl/queue/nr_requests : 128
Recommendation from agk@redhat.com is to raise dm-0 to something in the range 1,000-10,000 right away and right now it looks like the 128 on the individual devices isn't a bottleneck but we could safely raise those somewhat too.
Edited by Gregory Stark