Skip to content

Enable TLS support for Patroni API

Hossein Pursultani requested to merge 6261-patroni-api-tls into master

What does this MR do?

This MR:

  1. enables TLS on Patroni API server (see: https://patroni.readthedocs.io/en/latest/SETTINGS.html#rest-api)
  2. enables client TLS authentication in gitlab-ctl patroni ... and patronictl

Testing

Users and passwords

First run the following script to set up the required passwords (and their hash):

alias pg-password-md5="echo 'require \"openssl\"; puts Digest::MD5.hexdigest(\"#{ARGV[1]}#{ARGV[0]}\")' | ruby -- - "

export GITLAB_SQL_USER='gitlab'
export GITLAB_SQL_PASSWORD='secret'
export GITLAB_SQL_PASSWORD_HASH="$(pg-password-md5 ${GITLAB_SQL_USER} ${GITLAB_SQL_PASSWORD})"

export PGBOUNCER_SQL_USER='pgbouncer'
export PGBOUNCER_SQL_PASSWORD='secret'
export PGBOUNCER_SQL_PASSWORD_HASH="$(pg-password-md5 ${PGBOUNCER_SQL_USER} ${PGBOUNCER_SQL_PASSWORD})"

export REPLICATOR_SQL_USER='gitlab_replicator'
export REPLICATOR_SQL_PASSWORD='secret'
export REPLICATOR_SQL_PASSWORD_HASH="$(pg-password-md5 ${REPLICATOR_SQL_USER} ${REPLICATOR_SQL_PASSWORD})"

export CONSUL_USER='gitlab-consul'
export CONSUL_PASSWORD='secret'
export CONSUL_PASSWORD_HASH="$(pg-password-md5 ${CONSUL_USER} ${CONSUL_PASSWORD})"

Select build Docker image and replica counts

Use Docker EE image from the build pipeline:

# The registry requires authentication
export TEST_IMAGE='registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee'

# Update with the latest build tag
export TEST_TAG='14.1.1-rfbranch.345323818.3a65c338-0'

export CONSUL_REPLICA_COUNT=3
export PATRONI_REPLICA_COUNT=3

Generate TLS certificates

I use step which is simpler than openssl:

Note: Certificate SAN depends on the number of Consul and Patroni replicas for. Docker Compose generally allocates container IP addresses sequentially. The following script works under this assumption.

SAN="$(printf '--san 172.28.28.%s ' $(seq $((CONSUL_REPLICA_COUNT + 2)) $((CONSUL_REPLICA_COUNT + PATRONI_REPLICA_COUNT + 1))))"

mkdir -p ./tls/

echo 'secret' > tls/password.txt

step certificate create \
    --profile root-ca \
    --password-file tls/password.txt \
    patroni-ca tls/ca.crt.pem tls/ca.key.pem

step certificate create \
    --password-file tls/password.txt \
    --ca ./tls/ca.crt.pem \
    --ca-key ./tls/ca.key.pem \
    --ca-password-file tls/password.txt \
    --san patroni \
    --san localhost \
    --san 127.0.0.1 \
    $(echo $SAN) \
    patroni ./tls/patroni.crt.pem ./tls/patroni.key.pem

step certificate create \
    --ca ./tls/ca.crt.pem \
    --ca-key ./tls/ca.key.pem \
    --ca-password-file tls/password.txt \
    --insecure \
    --no-password \
    --san patroni \
    --san localhost \
    --san 127.0.0.1 \
    $(echo $SAN) \
    patroni ./tls/client.crt.pem ./tls/client.key.pem

# This is a terrible thing to do but required for testing:
chmod 755 ./tls
chmod 644 ./tls/*.pem

Run a test cluster

Use the following Docker Compose file and to set up Consul and Patroni services:

docker-compose -p gitlab up --scale patroni=${PATRONI_REPLICA_COUNT} --scale consul=${CONSUL_REPLICA_COUNT} -d
version: "3.9"

networks:
  test:
    driver: bridge
    ipam: 
      driver: default
      config: 
        - subnet: 172.28.28.0/24

services:
  consul:
    image: '${TEST_IMAGE}:${TEST_TAG}'
    networks:
      test:
    volumes: 
      - ${PWD}/tls:/opt/tls:ro
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        consul['enable'] = true

        consul['configuration'] = {
          server: true,
          client_addr: '0.0.0.0',
          bootstrap_expect: ${CONSUL_REPLICA_COUNT},
          retry_join: %w(consul)
        }

        gitlab_rails['auto_migrate'] = false
        
        gitlab_exporter['enable'] = false
        sidekiq['enable'] = false
        puma['enable'] = false
        registry['enable'] = false
        gitaly['enable'] = false
        gitlab_workhorse['enable'] = false
        nginx['enable'] = false
        prometheus_monitoring['enable'] = false
        redis['enable'] = false

  patroni:
    depends_on: 
      - consul
    image: '${TEST_IMAGE}:${TEST_TAG}'
    networks:
      test:
    volumes: 
      - ${PWD}/tls:/opt/tls:ro
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        roles ['patroni_role']

        postgresql['listen_address'] = '0.0.0.0'

        postgresql['pgbouncer_user_password'] = '${PGBOUNCER_SQL_PASSWORD_HASH}'
        postgresql['sql_user_password'] = '${GITLAB_SQL_PASSWORD_HASH}'
        postgresql['sql_replication_password'] = '${REPLICATOR_SQL_PASSWORD}'
        postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 172.28.28.0/24)
        postgresql['md5_auth_cidr_addresses'] = %w(127.0.0.1/32 172.28.28.0/24)

        consul['services'] = %w(postgresql)
        consul['configuration'] = {
          retry_join: %w(consul)
        }

        patroni['consul']['url'] = 'http://consul:8500'

        patroni['tls_verify'] = false
        patroni['tls_certificate_file'] = '/opt/tls/patroni.crt.pem'
        patroni['tls_key_file'] = '/opt/tls/patroni.key.pem'
        patroni['tls_key_password'] = 'secret'
        #patroni['tls_ca_file'] = '/opt/tls/ca.crt.pem'
        #patroni['tls_client_mode'] = 'required'
        #patroni['tls_client_certificate'] = '/opt/tls/client.crt.pem'
        #patroni['tls_client_key'] = '/opt/tls/client.key.pem'

        gitlab_rails['auto_migrate'] = false
        
        gitlab_exporter['enable'] = false
        sidekiq['enable'] = false
        puma['enable'] = false
        registry['enable'] = false
        gitaly['enable'] = false
        gitlab_workhorse['enable'] = false
        nginx['enable'] = false
        prometheus_monitoring['enable'] = false
        redis['enable'] = false

Test scenarios

Change the following configuration and test with gitlab-ctl patroni ... commands or curl https://patroni:8008 ....

patroni['tls_verify'] = false
patroni['tls_certificate_file'] = '/opt/tls/patroni.crt.pem'
patroni['tls_key_file'] = '/opt/tls/patroni.key.pem'
patroni['tls_key_password'] = 'secret'
#patroni['tls_ca_file'] = '/opt/tls/ca.crt.pem'
#patroni['tls_client_mode'] = 'required'
#patroni['tls_client_certificate'] = '/opt/tls/client.crt.pem'
#patroni['tls_client_key'] = '/opt/tls/client.key.pem'

Related issues

Closes #6261 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated: gitlab!67290 (merged)
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for the GitLab Chart opened
Edited by Hossein Pursultani

Merge request reports