Support Custom Certificate Authorities with Zoekt chart

The customer (internal link) has reported that Zoekt chart does not support configuring custom certificate authorities. It is required to support this for setups where Gitaly server is configured with TLS support behind a hostname signed by a custom certificate authority.

How to reproduce:

1. Install GitLab Helm chart

2. Create a Gitaly server with TLS support using a self-signed certificate:

• Generate self-signed certificate:

export gl_host=gitaly-server.example.com

openssl genrsa -des3 -out ca.key 4096 # qazwsxedc
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1024 -out ca.crt
openssl genrsa -out ${gl_host}.key 2048

openssl req -new -key ${gl_host}.key -out ${gl_host}.csr
openssl req -in ${gl_host}.csr -noout -text

openssl x509 -req -extfile <(printf "subjectAltName=DNS:${gl_host},DNS:www.${gl_host}") -in ${gl_host}.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ${gl_host}.crt -days 500 -sha256

cat ${gl_host}.crt ca.key > cert.pem
cp ${gl_host}.key > key.pem

• Upload the certificates and copy them to /etc/gitlab/ssl/:

# On Gitaly server:
sudo mkdir -p /etc/gitlab/ssl
sudo chmod 755 /etc/gitlab/ssl
sudo cp key.pem cert.pem /etc/gitlab/ssl/
sudo chmod 644 key.pem cert.pem

• Configure Gitaly server:

# Get the GitLab Shell secret should be used for the `shellsecret` value
kubectl get secret <release>-gitlab-shell-secret -ojsonpath='{.data.secret}' | base64 -d

# Get the Gitaly token, should be used for the `AUTH_TOKEN` value.
kubectl get secret <release>-gitaly-secret -ojsonpath='{.data.token}' | base64 -d

# Avoid running unnecessary services on the Gitaly server
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
puma['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitlab_exporter['enable'] = false
gitlab_kas['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false

# Prevent database connections during 'gitlab-ctl reconfigure'
gitlab_rails['auto_migrate'] = false

# Configure the gitlab-shell API callback URL. Without this, `git push` will
# fail. This can be your 'front door' GitLab URL or an internal load
# balancer.
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server.
gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'

gitaly['configuration'] = {
   # ...
   #
   # Make Gitaly accept connections on all network interfaces. You must use
   # firewalls to restrict access to this address/port.
   # Comment out following line if you only want to support TLS connections
   tls_listen_addr: '0.0.0.0:9999',
   tls: {
    certificate_path: '/etc/gitlab/ssl/cert.pem',
    key_path: '/etc/gitlab/ssl/key.pem',
   },
   auth: {
     # ...
     #
     # Authentication token to ensure only authorized servers can communicate with
     # Gitaly server
     token: 'AUTH_TOKEN',
   },
   storage: [
      {
         name: 'default',
         path: '/var/opt/gitlab/git-data/repositories',
      }
   ]
}

gitlab_shell['secret_token'] = 'shellsecret'

• Run gitlab-ctl reconfigure
• Make sure that external connections to port 9999 are allowed in firewall

3. Configure GitLab chart to use external Gitaly and use Zoekt:

• Create external-gitaly-token and custom-ca-certificate secret:

kubectl create secret generic external-gitaly-token --from-literal=token=AUTH_TOKEN
kubectl create secret generic custom-ca-certificate --from-file=gitaly-ca.crt=ca.crt

• Upgrade your GitLab helm deployment with these values:

certmanager-issuer:
  email: youruser@example.com
global:
  gitaly:
    enabled: false
    external:
      - name: default                   # required
        hostname: gitaly-server.example.com # required
        port: 9999                     # optional, default shown
        tlsEnabled: true
    authToken:
      secret: external-gitaly-token     # required
      key: token                        # optional, default shown
  certificates:
    customCAs:
      - secret: custom-ca-certificate
        keys:
          - gitaly-ca.crt
  hosts:
    domain: example.com
    externalIP: x.x.x.x
  minio:
    enabled: true
gitlab-zoekt:
  install: true
  replicas: 2
  indexStorage: 2Gi

4. Create a group and a project and complete the steps in Zoekt chart setup guide.

5. Committing code to the repository does not make that code searchable by Zoekt.

The following errros are visible in indexer logs:

time=2024-09-05T17:23:31.310Z level=INFO msg="IndexRepository start" project_id=3 force=false
time=2024-09-05T17:23:31.442Z level=INFO msg="IndexRepository end" project_id=3 force=false
2024/09/05 17:23:31 "POST http://indexer/indexer/index HTTP/1.0" from 127.0.0.1:38076 - 500 272B in 158.205921ms

Zoekt::IndexerWorker sidekiq worker reports this error:

Request failed with: #\u003cHTTParty::Response:0x1a4c30 parsed_response={\"success\"=\u003efalse, \"error\"=\u003e\"cannot find a default branch: rpc error: code = Unavailable desc = last connection error: connection error: desc = \\\"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\\\"\"}, @response=#\u003cNet::HTTPInternalServerError 500 Internal Server Error readbody=true\u003e, @headers={\"server\"=\u003e[\"nginx\"], \"date\"=\u003e[\"Thu, 29 Aug 2024 13:16:39 GMT\"], \"content-type\"=\u003e[\"application/json\"], \"content-length\"=\u003e[\"272\"], \"connection\"=\u003e[\"close\"]}\u003e","exception.backtrace":["ee/lib/gitlab/search/zoekt/client.rb:83:in `index'","ee/lib/gitlab/search/zoekt/client.rb:19:in `index'","ee/app/models/concerns/search/zoekt/searchable_repository.rb:15:in `update_zoekt_index!'","ee/app/workers/zoekt/indexer_worker.rb:38:in `block in perform'","lib/gitlab/exclusive_lease_helpers.rb:43:in