Investigate switching to Recyclable cache keys
Now that GitLab is running Rails 5.2: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/12145 (amazing work @engwan
This change arrived in Rails 5.2 via this change: https://github.com/rails/rails/pull/29092.
Heroku have done a great write-up on what recyclable cache keys are https://blog.heroku.com/cache-invalidation-rails-5-2-dalli-store#background-what-are-recyclable-cache-keys
How could this improve things?
GitLab, like most Rails applications, uses key-based cache expiration. This is a simple and powerful approach to avoiding invalidation by using a new cache key for each version of an entity. DHH blogged about it here: https://signalvnoise.com/posts/3113-how-key-based-cache-expiration-works
The downside to this approach is that old cache values are left in the cache, rather than being explicitly evicted. This can lead to a lot of churn, with valid values being evicted while invalidated cache values remain in the cache taking up space, but not being used.
With Rails 5.2, it's possible to switch to Recyclable Cache keys. To over simplify, un this setup, the key and the version and stored separately, meaning that each entity will have at most one cache value per entity, with versions overwriting one another.
This can be done with
config.active_record.cache_versioning = true
Switching to this approach could potentially lead to smaller caches being as effective as the existing cache (cc @ayufan and @craig-gomes for memory savings), but in the case of GitLab.com, could lead to a much better cache hit rate.
Obviously this approach is not without its risks, and we should investigate, but it could lead to performance improvements.
Related to https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30368
Impacted Service: redis-cache
Impacted Saturation Point: redis_memory