Manage gitlab-monitor in Redis HA mode

Closes #2409 (closed) #3866 (closed)

Problem

In Redis HA mode when sentinels are detected, we replace redis_url to use the master-group-name format, which gitlab-rails requires. However, gitlab-monitor, doesn't understand this format. We need to provide an option to specify a redis URL to gitlab-monitor. Also, it will be good to provide users an option to probe all sentinel nodes too.

1. gitlab-monitor isn't designed with sentinel nodes in mind, so it shouldn't be installed there.
2. If the application node doesn't have Redis, sidekiq probing should be disabled in the gitlab-monitor present in that node.
3. If needed, gitlab-monitor can be installed separately on Redis master and slave nodes and configured to probe them individually. However, only the one connected to master will succeed and others will fail silently. Also, we need to tell Prometheus to scrape all of those gitlab-monitor instances.

While option 3 seems ideal, it is not that straight forward to implement. In gitlab.com, we install gitlab-monitor from the tarball directly, not as an omnibus-gitlab package and configure it separately.

I think we can provide the following middle ground:

1. We disable sidekiq probing in HA mode by default.
2. We provide a configuration that users can use to force enable sidekiq probing (gitlab_monitor['sidekiq_probe_enable'])
3. If they are enabling sidekiq probing in HA mode, they are expected to provide gitlab_rails['redis_host'], gitlab_rails['redis_port'] and gitlab_rails['redis_password'] parameters (pointing to the master node).
4. We will no longer overwrite those with master_name (which is the original issue), but simply make gitlab-monitor use the provided values. The downside of this is, if the master goes down and a failover happens, the probe will be pointing to an absent redis instance.

Changes

• A new configuration options to handle gitlab-monitor in Redis HA mode
  • gitlab_monitor['probe_sidekiq'] = true to control if sidekiq probes should be done by gitlab-monitor or not.
    • Defaults to true in single-node installations
    • Defaults to false in Redis HA mode.
    • If user sets to true, all sentinel nodes will also be probed.
    • If set to true, expects gitlab_rails['redis_*'] settings to be provided
• Changes to how redis_url is computed
  • For gitlab-rails, gitlab-shell and gitlab-workhorse, master_name and master_port is used to compute redis_url in Sentinel mode
  • For gitlab-monitor, if probe_sidekiq is set, gitlab_rails['redis_*'] is used to compute redis_url
  • No longer overwriting Gitlab['gitlab_rails']['redis_*'] settings.