Skip to content

[since 16.6.0] Cache misses + ever increasing connections in redis cause high CPU

Summary

Since the 16.6 upgrade, we see a lot of cache misses (almost no hits) in our monitoring.

This causes very high CPU in our frontend nodes, high latencies, high DB CPU load and generally high response times for the end users.

Steps to reproduce

Unsure, we upgraded from 16.5 to 16.6

What is the current bug behavior?

High CPU on frontend nodes, since the 16.6. upgrade

What is the expected correct behavior?

Normal CPU and cache hits

Relevant logs

image.png

image.png

MicrosoftTeams-image

Made according to https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/tutorials/how_to_use_flamegraphs_for_perf_profiling.md#bonus-custom-capture-using-perf-record

flamegraph.svg

Details of package version

16.6.0

Edited by Max Wittig