feat: add support for Redis Sentinel passwords

What does this MR do?

This merge request introduces global.redis.sentinelAuth that will allow admins to create a Redis Sentinel password for all Sentinel instances. redisYmlOverride and instance-specific configuration cannot be used at the moment to reduce the complexity of testing and managing secrets.

GitLab Exporter

• This uses sentinel_password in the YAML config file.

GitLab Rails

• Workhorse: SentinelPassword in the config TOML
• GitLab Rails: sentinel_password in the various Redis YML files

KAS

• KAS reads the Redis Sentinel secret directly via the sentinel_password_file.

MailRoom

sentinel_username and sentinel_password can be specified as config parameters in gitlab-mail_room v0.0.25 (gitlab-org/ruby/gems/gitlab-mail_room!68 (merged)). The old way of passing the username and password involves adding a username and password field to each host sentinels. This requires gitlab-org/build/CNG!1847 (merged).

Related issues

Relates to #2902 (closed)

Follow-up items

• Add documentation. Need to confirm that defining a single sentinelAuth makes sense.
• Add support for Container Registry. Requires gitlab-org/container-registry!1642 (merged)

Testing

I created a separate Redis and Redis Sentinel deployment:

redis-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:latest
        ports:
        - containerPort: 6379
        env:
        - name: REDIS_PASSWORD
          value: "<password from gitlab-redis-secret secret>"
        command:
        - sh
        - -c
        - |
          echo "user default on >${REDIS_PASSWORD} ~* +@all allchannels" > /tmp/redis.conf
          redis-server /tmp/redis.conf
---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    app: redis

sentinel-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-sentinel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-sentinel
  template:
    metadata:
      labels:
        app: redis-sentinel
    spec:
      containers:
      - name: redis-sentinel
        image: bitnami/redis-sentinel:latest
        ports:
        - containerPort: 26379
        env:
        - name: REDIS_PASSWORD
          value: "<password from gitlab-redis-secret secret>"
        - name: SENTINEL_PASSWORD
          value: "my-sentinel-password"
        command:
        - sh
        - -c
        - |
          echo "Starting Redis Sentinel..."
          cat <<EOF > /opt/bitnami/redis-sentinel/etc/sentinel.conf
          sentinel monitor mymaster redis.default.svc.cluster.local 6379 2
          sentinel auth-pass mymaster ${REDIS_PASSWORD}
          sentinel resolve-hostnames yes
          requirepass ${SENTINEL_PASSWORD}
          EOF
          redis-sentinel /opt/bitnami/redis-sentinel/etc/sentinel.conf
---
apiVersion: v1
kind: Service
metadata:
  name: redis-sentinel
spec:
  ports:
  - port: 26379
    targetPort: 26379
  selector:
    app: redis-sentinel

Then I ran:

kubectl apply -f redis-deployment.yaml
kubectl apply -f sentinel-deployment.yaml

In values.yaml, I updated this:

gitlab:
  mailroom:
    image:
      pullPolicy: Always
      tag: sh-update-mailroom
global:
  redis:
    auth:
      enabled: true
      # secret:
      # key:
    host: mymaster
    # port: 6379
    # user: webservice
    sentinels:
      - host: 'redis-sentinel.default.svc.cluster.local'
        port: 26379
    sentinelAuth:
      enabled: true
      secret: gitlab-redis-sentinel-secret
      key: password

Then I created a secret:

kubectl create secret generic gitlab-redis-sentinel-secret --from-literal=password=my-sentinel-password

Then I deployed GitLab with MailRoom enabled and verified all the pods came up.

Author checklist

For general guidance, please follow our Contributing guide.

Required

For anything in this list which will not be completed, please provide a reason in the MR discussion.

• Merge Request Title and Description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• MR has a green pipeline.
• Documentation created/updated.
• Tests added/updated, and test plan for scenarios not covered by automated tests.
• Equivalent MR/issue for omnibus-gitlab opened.

Reviewers checklist

• MR has a green pipeline on https://gitlab.com/gitlab-org/charts/gitlab.
• Consider downstream impact to the Operator, as per evaluating impact from changes to GitLab Chart.

assigned to @stanhu

added featureaddition typefeature labels

	1 Warning
	Please check the QA job and compare with builds on master. If no new failures are reported in QA job, add QA:verified label.

1 Message

If this is a wider community contribution, the Community contribution label should be added. Once your MR is ready for review you can comment @gitlab-bot ready <@user> to request the first review to someone. It's recommended that you pick the one suggested by the reviewer roulette. But you can ignore it and assign it to someone else if you see fit. Merge requests are handled according to the workflow documented in our handbook and should receive a response within the limit documented in our First-response SLO. If you don't receive a response from the reviewer within the SLO, please mention @gitlab-org/distribution, or one of our Project Maintainers

Reviewer roulette

Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category and by a maintainer in all other categories.

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.

Reviewer	Maintainer
@dustinmm80 (UTC-5, 2 hours ahead of @stanhu)	@clemensbeck (UTC+2, 9 hours ahead of @stanhu)

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added 2 commits

• 1eca9e5f - feat: add support for Redis Sentinel passwords in GitLab Rails
• 19b1f56a - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

changed title from feat: add support for Redis Sentinel passwords in GitLab Exporter to feat: add support for Redis Sentinel passwords

changed the description

removed featureaddition typefeature labels

added featureaddition redis typefeature labels

added 2 commits

• 9420d8e5 - feat: add support for Redis Sentinel passwords in GitLab Rails
• e81f9620 - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

marked the checklist item Merge Request Title and Description are up to date, accurate, and descriptive. as completed

marked the checklist item MR targeting the appropriate branch. as completed

marked the checklist item MR has a green pipeline. as completed

marked the checklist item Tests added/updated, and test plan for scenarios not covered by automated tests. as completed

changed milestone to %17.1

added workflowready for review label

changed the description

added 2 commits

• 8a1e29be - feat: add support for Redis Sentinel passwords in GitLab Rails
• 31c6c127 - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

removed workflowready for review label

added 2 commits

• ef05df7c - feat: add support for Redis Sentinel passwords in GitLab Rails
• b9b3e758 - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

added workflowready for review label

added 2 commits

• 556d4ee6 - feat: add support for Redis Sentinel passwords in GitLab Rails
• f532312b - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

added 4 commits

• 06427b41 - Drop extraneous map check in Redis Sentinel template
• e5fef185 - feat: add support for Redis Sentinel passwords in KAS
• acc08b3f - feat: add support for Redis Sentinel passwords in GitLab Rails
• 83f213ce - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

resolved all threads

added 4 commits

• c85b5d64 - feat: add support for Redis Sentinel passwords in GitLab Exporter
• ecac7a44 - feat: add support for Redis Sentinel passwords in KAS
• 4e114855 - feat: add support for Redis Sentinel passwords in GitLab Rails
• 518467bc - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

added 3 commits

• 84d68f3d - feat: add support for Redis Sentinel passwords in KAS
• b56e0925 - feat: add support for Redis Sentinel passwords in GitLab Rails
• 21a833c4 - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

added 4 commits

• 5b3e17dc - feat: add support for Redis Sentinel passwords in GitLab Exporter
• 7c0d9eaf - feat: add support for Redis Sentinel passwords in KAS
• 8d786616 - feat: add support for Redis Sentinel passwords in GitLab Rails
• 17a7cdcf - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

requested review from @Alexand

resolved all threads

added 1 commit

• 79d88ada - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

changed the description

Need to confirm that defining a single sentinelAuth makes sense

Do you mean as a global config? I think it does. If I'm setting up sentinel, I'm probably interested in having high availability for all my services that use Redis. Is there a great benefit in configuring just for some? Maybe having HA for just some of them might be a cost cutting choice? I'm not sure if it would be a considerable cut though.

I think we can probably also start with global, then later introduce per sub-chart overrides if people start asking for it.

Global also keeps a smaller config.

For this decision, it's a good idea to ask @WarheadsSE to weigh-in. There's a conference going on, so maybe he'll answer when he comes back from it sometime next week.

But yeah, let's push the review with the current proposed approach for now.

added 2 commits

• e070407a - feat: add support for Redis Sentinel passwords in GitLab Rails
• f7312d83 - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

added 1 commit

• 01fbcfa3 - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

added devopssystems groupdistribution group::distributionbuild labels

added sectioncore platform label

mentioned in merge request gitlab-org/build/CNG!1847 (merged)

mentioned in issue gitlab-org/container-registry#1269 (closed)

mentioned in issue #2902 (closed)

added 10 commits

• 01fbcfa3...60f7e35e - 6 commits from branch master
• 4b097809 - feat: add support for Redis Sentinel passwords in GitLab Exporter
• ca7023c6 - feat: add support for Redis Sentinel passwords in KAS
• 29f970d8 - feat: add support for Redis Sentinel passwords in GitLab Rails
• 676d8114 - feat: add support for Redis Sentinel passwords in MailRoom

Compare with previous version

KAS

The gitlab-kas pods are up but are continually reporting:

{"level":"error","time":"2024-06-06T19:45:42.762Z","msg":"Error handling Redis SUBSCRIBE","error":"NOPERM No permissions to access a channel"}

Could be a problem on KAS sentinel support, so not necessarily this configs change. But needs investigating.

changed the description

mentioned in issue #3813 (closed)

Mailroom

I've re-tested all and it looks like it's all in good shape.

We also need some special config for mailroom. I've set it up with a real GMail account like so:

global:
  appConfig:
    incomingEmail:
      enabled: true
      # This '+%{key}' suffix should be included in its entirety within the email
      # address and not replaced by another value.
      address: <REDACTED>+%{key}@gmail.com
      host: imap.gmail.com
      port: 993
      ssl: true
      startTls: false
      user: <REDACTED>@gmail.com
      password:
        secret: incoming-email-password
        key: secret
      mailbox: inbox
      deliveryMethod: webhook
      authToken: {}

Also had to create the incoming-email-password secret with my IMAP account application password.

I got surprised when I cat /var/opt/gitlab/mail_room.yml and saw sentinel_password: "<%= File.read("/etc/gitlab/redis-sentinel/redis-sentinel-password").strip %>". I didn't know it was an ERB friendly file, which I later confirmed.

I've then checked the logs and connection worked well with the IMAP server.

---

Exporter

Running curl localhost:9168/metrics returned plenty of metrics.

KAS and Rails

I did another more complete test by going into the instance, configuring an agent and using the CI/CD feature.

Configs and logs

I've verified all configs and logs, and I couldn't find anything unexpected.

approved this merge request

added workflowin review label and removed workflowready for review label

requested review from @WarheadsSE