Add TLS support configuration to Workhorse

What does this MR do?

Note: This MR requires changes from gitlab-org/build/CNG!1057 (merged) and the two must be merged together othrwise autodeploy will deploy Workhorse without metrics on GitLab.com. See: gitlab-org/build/CNG!1057 (comment 1018128850)

This MR adds TLS support configuration for Workhorse listeners, including its monitoring listener.

Testing

1. Create a TLS certificate and key. Assuming that you have cert-manager.io installed on your cluster you can use the following to generate a self-signed certificate:

   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
     name: selfsigned-issuer
   spec:
     selfSigned: {}
   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: workhorse-selfsigned-cert
   spec:
     isCA: true
     commonName: RELEASE-webservice-default.NAMESPACE.svc
     dnsNames:
     - localhost
     secretName: gitlab-workhorse-tls
     issuerRef:
       name: selfsigned-issuer
       kind: Issuer
       group: cert-manager.io

   Replace RELEASE and NAMESPACE with your values. This will create a TLS Secret named workhorse-selfsigned-cert.

2. Once workhorse-selfsigned-cert is created, create another Secret, named workhorse-selfsigned-ca, with ca.crt key and use tls.crt value of workhorse-selfsigned-cert.

3. Use the following values to configure Workhorse TLS support:

   global:
     workhorse:
       tls:
         enabled: true
     certificates:
       customCAs:
         secret: workhorse-selfsigned-ca
   
   gitlab:
     webservice:
       workhorse:
         tls:
           secretName: workhorse-selfsigned-cert
           caSecretName: workhorse-selfsigned-ca
         monitoring:
           exporter:
             enabled: true
             tls:
               enabled: true
   

   This will enable TLS on both web and monitoring listener and verifies the TLS certificate as well.

4. Verify by accessing the instance, try to do a Git pull or push to trigger Gitaly, use SSH endpoint (e.g. ssh -T git@gitlab.DOMAIN) to trigger GitLab Shell. Check Nginx Ingress, Gitaly, and GitLab Shell logs.

Related issues

Closes #3316 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• Merge Request Title and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for omnibus-gitlab opened