Investigate steady increase in redis persistent memory usage
Initially raised in https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2471#note_1568594676, we are seeing a steady increase of redis-persistent memory usage roughly at a rate of ~100 MB/week, starting from the week of **2023-08-20**. Note the increase in the weekend's baseline below. ![image](/uploads/2d6816d0b914596d746c96144a11eb8b/image.png) [source](https://thanos.gitlab.net/graph?g0.expr=redis_memory_used_rss_bytes%7Benv%3D%22gprd%22%2C%20type%3D%22redis%22%7D%20and%20on%20(instance)%20redis_instance_info%7Brole%3D%22master%22%7D&g0.tab=0&g0.stacked=0&g0.range_input=10w&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2023-10-03%2003%3A25%3A26&g0.moment_input=2023-10-03%2003%3A25%3A26) This issue aims to investigate and mitigate the root cause of the growth. We are not in any saturation event in the near future, especially once we've moved this instance to [Redis Cluster](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1055) with [bigger capacity](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2471#note_1564839284), but it'd be preferable to address this early.
issue