Skip to content

Add configurable startup delay to Redis

Jacob Vosmaer requested to merge jv-redis-startup-delay into master

What does this MR do?

This MR adds a configurable startup delay to Redis. By default the delay is 0s.

The intended use is to force a Sentinel failover in certain conditions. After a Redis restart, its database contents are either gone (when persistence is disabled) or stale (because RDB dumps happen periodically). This means that you would almost always want a Redis restart to force a Sentinel failover. This is not what happens in practice; the way we have Sentinel configured on GitLab.com, it can happen that a Redis process restarts too fast for Sentinel to notice. When that happens we either lose the entire Redis database, or end up resetting it to the most recent RDB dump.

By injecting a startup delay, we can make Redis restarts slow, forcing a failover.

Related issues

gitlab-com/gl-infra/scalability#2153 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks N/A
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated We first need to learn how to use this on GitLab.com
  • Tests added
  • Integration tests added to GitLab QA Redis supervision is not testable with GitLab QA
  • Equivalent MR/issue for the GitLab Chart opened GitLab Chart does not manage Redis
Edited by Jacob Vosmaer

Merge request reports