Skip to content

Support running Redis and Sentinel only over SSL

Balasankar 'Balu' C requested to merge 6627-sentinel-behind-tls-only into master

What does this MR do?

Support running Redis and Sentinel only over SSL, without having a non-TLS port open.

Also expands the gitlab_rails['sentinels'] setting to hold more details - like whether connection should be over SSL or not, client certificate details, etc.

Testing

  1. I used the following docker-compose.yml file to spin up a minimal instance, with only TLS ports open.

    docker-compose.yml 
    version: '3.7'

x-default_settings: &default_settings
  image: "${GITLAB_IMAGE}"
  privileged: true
  shm_size: '256m'
  restart: always
  volumes:
    - "./ssl:/etc/gitlab/ssl"
    - "./trusted-certs:/etc/gitlab/trusted-certs"

services:
  redis-primary-sentinel:
    <<: *default_settings
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        roles ['redis_sentinel_role', 'redis_master_role']

        # Redis configuration
        redis['bind'] = '10.0.0.2'
        redis['tls_port'] = 6380
        redis['tls_cert_file'] = '/etc/gitlab/ssl/self_signed.crt'
        redis['tls_key_file'] = '/etc/gitlab/ssl/self_signed.key'
        redis['tls_replication'] = 'yes'
        redis['password'] = 'toomanysecrets'

        # Sentinel Configuration
        sentinel['bind'] = '10.0.0.2'
        sentinel['quorum'] = 2
        sentinel['tls_port'] = 26380
        sentinel['tls_cert_file'] = '/etc/gitlab/ssl/self_signed.crt'
        sentinel['tls_key_file'] = '/etc/gitlab/ssl/self_signed.key'
        sentinel['tls_replication'] = 'yes'

        # Information about the primary
        redis['master_ip'] = '10.0.0.2'
        redis['master_port'] = '6380'
        redis['master_name'] = 'gitlab-redis'
        redis['master_password'] = 'toomanysecrets'

        # Disable Redis and Sentinel non-TLS ports
        redis['port'] = 0
        sentinel['port'] = 0

        # This is a pure Redis node. We don't need Rails stuff.
        gitlab_rails['enable'] = false
    networks:
      default:
        ipv4_address: 10.0.0.2

  redis-replica-1-sentinel:
    <<: *default_settings
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        roles ['redis_sentinel_role', 'redis_replica_role']

        # Redis configuration
        redis['bind'] = '10.0.0.3'
        redis['tls_port'] = 6380
        redis['tls_cert_file'] = '/etc/gitlab/ssl/self_signed.crt'
        redis['tls_key_file'] = '/etc/gitlab/ssl/self_signed.key'
        redis['password'] = 'toomanysecrets'
        redis['tls_replication'] = 'yes'

        # Sentinel configuration
        sentinel['bind'] = '10.0.0.3'
        sentinel['quorum'] = 2
        sentinel['tls_port'] = 26380
        sentinel['tls_cert_file'] = '/etc/gitlab/ssl/self_signed.crt'
        sentinel['tls_key_file'] = '/etc/gitlab/ssl/self_signed.key'
        sentinel['tls_replication'] = 'yes'

        # Information about the primary
        redis['master_name'] = 'gitlab-redis'
        redis['master_password'] = 'toomanysecrets'
        redis['master_ip'] = '10.0.0.2'
        redis['master_port'] = '6380'

        # Disable Redis and Sentinel non-TLS ports
        redis['port'] = 0
        sentinel['port'] = 0

        # This is a pure Redis node. We don't need Rails stuff.
        gitlab_rails['enable'] = false
    networks:
      default:
        ipv4_address: 10.0.0.3

  redis-replica-2-sentinel:
    <<: *default_settings
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        roles ['redis_sentinel_role', 'redis_replica_role']

        # Redis configuration
        redis['bind'] = '10.0.0.4'
        redis['tls_port'] = 6380
        redis['tls_cert_file'] = '/etc/gitlab/ssl/self_signed.crt'
        redis['tls_key_file'] = '/etc/gitlab/ssl/self_signed.key'
        redis['tls_replication'] = 'yes'
        redis['password'] = 'toomanysecrets'

        # Sentinel configuration
        sentinel['bind'] = '10.0.0.4'
        sentinel['quorum'] = 2
        sentinel['tls_port'] = 26380
        sentinel['tls_cert_file'] = '/etc/gitlab/ssl/self_signed.crt'
        sentinel['tls_key_file'] = '/etc/gitlab/ssl/self_signed.key'
        sentinel['tls_replication'] = 'yes'

        # Information about the primary
        redis['master_name'] = 'gitlab-redis'
        redis['master_password'] = 'toomanysecrets'
        redis['master_ip'] = '10.0.0.2'
        redis['master_port'] = '6380'

        # Disable Redis and Sentinel non-TLS ports
        redis['port'] = 0
        sentinel['port'] = 0

        # This is a pure Redis node. We don't need Rails stuff.
        gitlab_rails['enable'] = false
    networks:
      default:
        ipv4_address: 10.0.0.4

  gitlab:
    <<: *default_settings
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url "http://127.0.0.1"

        # Redis settings
        redis['enable'] = false
        redis['master_name'] = 'gitlab-redis'
        redis['master_password'] = 'toomanysecrets'
        gitlab_rails['redis_sentinels'] = [
          {'host' => '10.0.0.2', 'port' => 26380, 'ssl' => true},
          {'host' => '10.0.0.3', 'port' => 26380, 'ssl' => true},
          {'host' => '10.0.0.4', 'port' => 26380, 'ssl' => true}
        ]
        gitlab_rails['redis_ssl'] = true
    ports:
      - "80:80"
      - "443:443"
      - "22:22"
    networks:
      default:
        ipv4_address: 10.0.0.5

networks:
  default:
    ipam:
      config:
        - subnet: 10.0.0.0/24

  2. Confirmed that the master is detected correctly, and the TLS port is specified (thus covering #6628 (closed))

    $ docker-compose exec -it gitlab gitlab-ctl get-redis-master
Redis master found at host 10.0.0.2 listening on port 6380

  3. Confirmed that failover worked fine

    $ docker-compose exec -it redis-primary-sentinel gitlab-ctl stop redis
$ sleep 10
$ docker-compose exec -it gitlab gitlab-ctl get-redis-master
Redis master found at host 10.0.0.4 listening on port 6380

  4. Confirmed that second reconfigure still works on a sentinel node (thus covering #6627 (closed))

    $ docker-compose exec -it redis-replica-2-sentinel gitlab-ctl reconfigure

Related issues

Closes #6627 (closed)

Closes #6628 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • MR title and description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • Latest Merge Result pipeline is green.
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

  • The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
  • If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
  • If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes.
  • Documentation created/updated.
  • Tests added.
  • Integration tests added to GitLab QA.
  • Equivalent MR/issue for the GitLab Chart opened.
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.
Edited by Balasankar 'Balu' C

Merge request reports