Add TLS support for KAS redis

What does this MR do?

Enables TLS in the KAS redis config file when the scheme in the Helm values is rediss.

Related issues

Closes #2921 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• Merge Request Title and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• [-] Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for omnibus-gitlab opened
  • This was already done in gitlab-org/omnibus-gitlab!5632 (merged)

Test plan

1. Run a self-signed redis
2. Run a local kind cluster and install GitLab
3. Install agentk to force KAS to connect to the Redis

Note: This guide uses direct paths to Redis and OpenSSL installed using Homebrew on macOS. I spelled those out because brew did not install a symlink in /usr/local/bin. redis-cli and openssl. On other systems, can presumably use redis-server and openssl directly. Redis must be version 6 or later, and OpenSSL recent enough to include the -addext flag.

1. Run a self-signed redis

Set up a loopback alias (assume it is 172.16.123.1). We will use this for the self-signed redis.

Create a certificate (redis.crt) and key (redis.key):

/usr/local/opt/openssl/bin/openssl req -new -x509 \
    -days 365 \
    -newkey rsa:2048 \
    -nodes \
    -subj "/CN=172.16.123.1/" \
    -addext "subjectAltName = IP:172.16.123.1" \
    -keyout redis.key \
    -out redis.crt

In a separate tab, start redis:

/usr/local/opt/redis/bin/redis-server --tls-port 6379 --port 0 \
  --tls-cert-file redis.crt \
  --tls-key-file redis.key \
  --tls-ca-cert-file redis.crt \
  --tls-auth-clients no \
  --requirepass hello-redis

2. Run a cluster and install GitLab

Create a kind cluster:

kind create cluster

Create secrets:

kubectl create secret generic redis-ca --from-file="redis.crt"
kubectl create secret generic redis-password --from-literal=redis-password=hello-redis

Create test-values.yaml:

# test-values.yaml
global:
  kas:
    enabled: true
  redis:
    host: 172.16.123.1
    port: 6379
    password:
      enabled: true
      secret: redis-password
      key: redis-password
    scheme: rediss
  certificates:
    customCAs:
      - secret: redis-ca
redis:
  install: false

Install the chart

helm upgrade --install gitlab . \
  --set global.hosts.domain=172.16.123.1 \
  -f examples/kind/values-base.yaml \
  -f examples/kind/values-no-ssl.yaml \
  -f test-values.yaml

Note: For the purpose of this test, as long as gitlab-kas starts, we don't care about the other pods.

3. Install agentk to force KAS to connect to the Redis

As a shortcut, we use a dummy token because this is enough to trigger the redis connection:

docker run --pull=always \
       --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable \
       generate \
       --agent-token=your-agent-token \
       --kas-address=ws://gitlab-kas.default.svc.cluster.local:8150 \
       --agent-version=v14.3.2 \
       --namespace gitlab-kubernetes-agent | kubectl apply -f -

In the gitlab-kas logs, you should now see logs such as

{"level":"error","time":"2021-10-07T20:27:40.715Z","msg":"AgentInfo()","correlation_id":"01FHE62J39KM33KX3NAVQKHQCM","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"HTTP status code: 500"}     

The key is that we're seeing HTTP errors from AgentInfo(). This means that the rate limiter was already engaged and that it is trying to talk to GitLab. In my case, my GitLab webservice never came live due to resource constraints (insufficient memory and CPU), so I was getting 5xx errors, but otherwise we would expect 4xx errors.