[since 16.6.0] Cache misses + ever increasing connections in redis cause high CPU
Summary
Since the 16.6 upgrade, we see a lot of cache misses (almost no hits) in our monitoring.
This causes very high CPU in our frontend nodes, high latencies, high DB CPU load and generally high response times for the end users.
Steps to reproduce
Unsure, we upgraded from 16.5 to 16.6
What is the current bug behavior?
High CPU on frontend nodes, since the 16.6. upgrade
What is the expected correct behavior?
Normal CPU and cache hits
Relevant logs
Made according to https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/tutorials/how_to_use_flamegraphs_for_perf_profiling.md#bonus-custom-capture-using-perf-record
Details of package version
16.6.0
Edited by Max Wittig