Provision new Sidekiq shard's Redis instance in gprd

1. Secrets setup

We will need more secrets than usual since we have 2 extra users: sentinel and gitlab_monitor.

for I in SENTINEL_REDACTED SK_EXPORTER_REDACTED REPLICA_REDACTED RAILS_REDACTED EXPORTER_REDACTED CONSOLE_REDACTED; do echo $I; openssl rand -hex 32; done

a. redis vault

Before running the following, we may need to create the vault using gkms-vault-create.

./bin/gkms-vault-create redis gprd

{
  "redis-sidekiq-catchall-a": {
    "sentinel_conf": {
      "user": "sentinel",
      "password": "<SENTINEL_REDACTED>"
    },
    "redis_conf": {
      "masteruser": "replica",
      "masterauth": "<REPLICA_REDACTED>",
      "user": [
        "default off",
        "replica on ~* &* +@all ><REPLICA_REDACTED>",
        "sentinel on ~* &* +@all ><SENTINEL_REDACTED>",
        "console on ~* &* +@all ><CONSOLE_REDACTED>",
        "redis_exporter on +client +ping +info +config|get +cluster|info +slowlog +latency +memory +select +get +scan +xinfo +type +pfcount +strlen +llen +scard +zcard +hlen +xlen +eval allkeys ><EXPORTER_REDACTED>",
        "gitlab_monitor on +client +ping +info +config|get +info +slowlog +select +get +scan +zscan +sscan +type +hget +hmget +hgetall +exists +zrangebyscore +zrange +zcount +llen +lrange +lindex +scard +zcard allkeys ><SK_EXPORTER_REDACTED>",
        "rails on ~* &* +@all -debug ><RAILS_REDACTED>"
      ]
    }
  }
}

Note that gitlab_monitor allowed read commands are obtained from:

• https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/api.rb
• https://gitlab.com/gitlab-org/ruby/gems/gitlab-exporter/-/blob/master/lib/gitlab_exporter/sidekiq.rb

b. redis-exporter vault

./bin/gkms-vault-edit redis-exporter gprd

...
{
  "redis_exporter": {
    "redis-sidekiq-catchall-a": {
      "env": {
        "REDIS_PASSWORD": "<EXPORTER_REDACTED>"
      }
    }
  }
}

c. omnibus secrets vault

gitlab monitor credentials needs to be loaded into gitlab-omnibus-secrets in this structure (referencing https://gitlab.com/gitlab-cookbooks/gitlab-monitor/-/blob/master/recipes/redis.rb#L18)

Use glsh vault edit-secret chef env/gprd/shared/gitlab-omnibus-secrets and add the redis-sidekiq-catchall-a key in the following schema: omnibus-gitlab.gitlab_rb (note that there will be other keys present, the snippet below is simplified).

"omnibus-gitlab": {
  ...
  "gitlab_rb": {
    ...
    "redis-sidekiq-catchall-a": {
      "password": <SK_EXPORTER_REDACTED>
    }
  }
}

2. Chef roles setup

The chef-roles will need to include the gitlab-monitor::redis recipe.

• https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/4574

3. VM provisioning

This is fairly straightforward and can be done on config-mgmt. https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/8061

4. Assigning replicas and masters

Referencing #2049 (comment 1202287292), we may need to set the replicaof.

We could create a script for runbook which takes in a list of hosts and:

1. check for a 1 master - n replica topology
2. set replicaof if (1) is false.