vmstat: strange iowait values
Hi,
We've noticed something weird with the values returned by vmstat for the iowait. It's the worst kind of weirdness: it happens randomly. The symptoms are simple enough: sometimes the iowait values are just insanely high. Ex.:
$ vmstat -n 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 234752 2591036 2522716 26450904 0 0 132 222 4 3 0 0 99 1 0
0 1 234752 2538500 2522716 26503648 0 0 26652 0 506 556 0 0 97 2 0
0 1 234752 2488484 2522716 26552964 0 0 22828 0 480 562 0 0 101 870819400 0
$ vmstat -n 1 4
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 299520 243300 1017816 27957512 0 0 49 109 0 0 0 0 100 0 0
0 1 299520 286188 1017848 27831868 0 192 147796 322 1001 615 3 1 96 0 0
0 1 299776 302512 1017856 27762216 0 116 97784 116 564 339 2 1 96 2 0
0 1 299776 302624 1017860 27736684 0 56 42660 56 616 333 1 0 99 870819401 0
- The erronous value is always on the last tic.
- We've witnessed this on different systems and versions: Debian 8, 9, 10, 11 and CentOS 7.6, on both physical and virtual machines.
- It can happen anytime during the day, there's no recurring pattern. And when it happens, the load of the machines is pretty normal.
Crude bash script to capture occurences (run it with a cron every minutes):
#!/bin/bash
for number in {1..10}; do
stats=$(vmstat -n 1 4)
line=$( echo "$stats" | tail -1 )
iowait=$( echo "$line" | awk '{print $16}' )
if [ "$iowait" -gt "100" ]; then
echo "$( date +'%Y-%m-%dT%H:%M:%S.%6N%:z') $stats" >> /var/log/vmstats.log
fi
sleep 2
done
Versions:
- Debian 8: vmstat from procps-ng 3.3.9
- Debian 9: vmstat from procps-ng 3.3.12
- Debian 10: vmstat from procps-ng 3.3.15
- Debian 11: vmstat from procps-ng 3.3.17
- CentOS 7.6 : vmstat from procps-ng 3.3.10