NUMA-hinted page migrations pushing redis main thread to CPU saturation on redis-cache

Redis-cache main thread CPU saturation starting on 2022-08-26

The redis main thread on the primary redis-cache instance (currently redis-cache-02) unexpectedly hit CPU saturation today.

This turns out to be due to foreground page migrations between NUMA nodes, an overhead incurred on the redis main thread during page faults injected by kernel's NUMA balancing mechanism.

Initial notes

For context:

The CPU spikes started today at around 2022-08-26 07:00 UTC, and they continued to occur intermittently throughout the normal daily rise in weekday traffic.
These saturation events correlate with large spikes in client connection count, which could potentially be either a cause or an effect of redis's CPU usage spike.
- The connection count spikes appear to correspond with deploys, but the size of the connection spikes is much larger than usual: 50% increase (e.g. abruptly jumping from 14K to 22K connections).
Yesterday we upgraded these VMs, doubling their memory and CPU capacity: production#7627. Implicitly that also means we:
- Booted to a new kernel release (but not a new redis release).
- Probably migrated to a different physical machine with different neighbors.
- Potentially have a larger number of hyperthreads per physical core. If so, this might affect the scheduler. (We do not want the redis main thread to have to compete on the same core with other CPU-greedy tasks, such as redis IO threads.)
- Lost any unmanaged kernel tuning adjustments. (None expected, just mentioning for completeness.)

Status 2022-08-26

The new CPU overhead appears to be kernel doing foreground page migrations between NUMA nodes during page faults by the redis main thread.

Next steps include capturing more profiling data to check this conclusion, and reducing the new overhead by either kernel tuning or downgrading the machine-type.