top cpu usage shows as 0% despite spinning process
I've encountered an issue when running top that results in a particular CPU core showing as 100% idle despite a process spinning on that core (turbostat, for example, shows 100% busy). It only happens on some cores. I have confirmed it occurs with my distribution top (procps-ng 3.3.10) as well as with 3.3.15 built from source and master built from git source.
You can see in the below that cpu 10, 11 and 13 are all reporting "100% idle" in top, but "100% busy" in turbostat. In reality, they are 100% busy, the same as the other cores 6+.
I'm running CentOS7 with kernel 4.19.43 (this behavior was not present on 4.14.71, or on the original CentOS kernel).
Here's the output from top:
top - 13:19:54 up 21:39, 3 users, load average: 15.47, 15.61, 15.46
Tasks: 274 total, 2 running, 272 sleeping, 0 stopped, 0 zombie
%Cpu0 : 11.1 us, 15.6 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 13.9 us, 4.1 sy, 0.0 ni, 71.9 id, 0.0 wa, 0.0 hi, 10.1 si, 0.0 st
%Cpu2 : 16.2 us, 4.0 sy, 0.0 ni, 75.5 id, 0.0 wa, 0.0 hi, 4.3 si, 0.0 st
%Cpu3 : 14.4 us, 4.6 sy, 0.0 ni, 80.6 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
%Cpu4 : 12.1 us, 5.9 sy, 0.0 ni, 70.7 id, 0.0 wa, 0.0 hi, 11.4 si, 0.0 st
%Cpu5 : 17.0 us, 3.8 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.0 hi, 3.4 si, 0.0 st
%Cpu6 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu12 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu13 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu14 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu15 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu16 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu17 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu18 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu19 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 stHere's the output from turbostat:
Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI CPU%c1 CPU%c3 CPU%c6 CoreTmp PkgTmp PkgWatt RAMWatt PKG_% RAM_%
- - - 3097 81.69 3700 3177 802894 0 18.31 0.00 0.00 48 48 181.94 0.00 0.00 0.00
0 0 0 1452 38.29 3700 3177 243066 0 61.71 0.00 0.00 35 42 78.07 0.00 0.00 0.00
0 1 1 1773 46.76 3700 3177 137996 0 53.24 0.00 0.00 35
0 2 2 1436 37.88 3700 3177 104214 0 62.12 0.00 0.00 36
0 3 3 1143 30.14 3700 3177 75248 0 69.86 0.00 0.00 35
0 4 4 1516 39.99 3700 3177 128367 0 60.01 0.00 0.00 35
0 8 5 1544 40.71 3700 3177 113914 0 59.29 0.00 0.00 35
0 9 6 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 41
0 10 7 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 42
0 11 8 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 41
0 12 9 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 41
1 0 10 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 41 48 103.88 0.00 0.00 0.00
1 1 11 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 45
1 2 12 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 47
1 3 13 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 48
1 4 14 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 45
1 8 15 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 43
1 9 16 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 44
1 10 17 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 42
1 11 18 3791 100.00 3700 3177 5 0 0.00 0.00 0.00 42
1 12 19 3791 100.00 3700 3177 24 0 0.00 0.00 0.00 41