Add configurable startup delay to Redis
What does this MR do?
This MR adds a configurable startup delay to Redis. By default the delay is 0s.
The intended use is to force a Sentinel failover in certain conditions. After a Redis restart, its database contents are either gone (when persistence is disabled) or stale (because RDB dumps happen periodically). This means that you would almost always want a Redis restart to force a Sentinel failover. This is not what happens in practice; the way we have Sentinel configured on GitLab.com, it can happen that a Redis process restarts too fast for Sentinel to notice. When that happens we either lose the entire Redis database, or end up resetting it to the most recent RDB dump.
By injecting a startup delay, we can make Redis restarts slow, forcing a failover.
Related issues
gitlab-com/gl-infra/scalability#2153 (closed)
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion
Required
-
Merge Request Title, and Description are up to date, accurate, and descriptive -
MR targeting the appropriate branch -
MR has a green pipeline on GitLab.com -
Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooksN/A -
trigger-package
has a green pipeline running against latest commit
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes -
Documentation created/updatedWe first need to learn how to use this on GitLab.com -
Tests added -
Integration tests added to GitLab QARedis supervision is not testable with GitLab QA -
Equivalent MR/issue for the GitLab Chart openedGitLab Chart does not manage Redis