Skip to content

E2E: Add logging of environment certs for mtls scenario

John McDonnell requested to merge jmd/debug-mtls-failures into master

What does this MR do and why?

E2E: Add logging of environment certs for mtls scenario

#431474 (comment 1891541780) notes that some failures can occur in the MTLS scenario due to TLS error when gitaly tries to access the /internal/allowed endpoints. In order to debug this, let's first verify what certs are actually present on disk to determine if the flakiness is arising from the test environment config or from an application config perspective.

I've been unable to find a way to reproduce these errors, but they do continue to occur so the hope will be that this will be a first step in helping to confirm/disprove a theory that something maybe going wrong with loading the certs into the test environment itself.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

Verify the output of the mtls job in the e2e:package-and-test-ee pipeline contains the additional logs presenting the list of certs on disk.

The log itself can be found in gitlab-qa-run-.../gitlab-ee-qa-.../qa-tests.log e.g. https://gitlab.com/gitlab-org/gitlab/-/jobs/7204817324/artifacts/browse/gitlab-qa-run-2024-06-27-16-20-08-78367a37/gitlab-ee-qa-d6b003dc/

Sample Output
[Jun 27 2024 16:26:21 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab bash -c 'ls -al /etc/gitlab/trusted-certs'`
[Jun 27 2024 16:26:21 UTC (QA Tests)] DEBUG -- Command output:
total 24
drwxr-xr-x 2 root root 4096 Jun 27 16:21 .
drwxrwxr-x 4 root root 4096 Jun 27 16:21 ..
lrwxrwxrwx 1 root root    6 Jun 27 16:21 120aa788.0 -> ca.crt
-rw-r--r-- 1 root root 1984 Jun 27 16:20 ca.crt
-rw-r--r-- 1 root root 3243 Jun 27 16:20 ca.key
-rw-r--r-- 1 root root 5227 Jun 27 16:20 ca.pem
[Jun 27 2024 16:26:21 UTC (QA Tests)] INFO  -- Executing: `docker exec gitlab bash -c 'ls -al /etc/gitlab/ssl'`
[Jun 27 2024 16:26:21 UTC (QA Tests)] DEBUG -- Command output:
total 32
drwxr-xr-x 2 root root 4096 Jun 27 16:20 .
drwxrwxr-x 4 root root 4096 Jun 27 16:21 ..
-rw-r--r-- 1 root root 1911 Jun 27 16:20 gitaly.test.crt
-rw-r--r-- 1 root root 1704 Jun 27 16:20 gitaly.test.csr
-rw-r--r-- 1 root root 3247 Jun 27 16:20 gitaly.test.key
-rw-r--r-- 1 root root 1911 Jun 27 16:20 gitlab.test.crt
-rw-r--r-- 1 root root 1704 Jun 27 16:20 gitlab.test.csr
-rw-r--r-- 1 root root 3243 Jun 27 16:20 gitlab.test.key
[Jun 27 2024 16:26:21 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly bash -c 'ls -al /etc/gitlab/trusted-certs'`
[Jun 27 2024 16:26:21 UTC (QA Tests)] DEBUG -- Command output:
total 24
drwxr-xr-x 2 root root 4096 Jun 27 16:21 .
drwxrwxr-x 4 root root 4096 Jun 27 16:21 ..
lrwxrwxrwx 1 root root    6 Jun 27 16:21 120aa788.0 -> ca.crt
-rw-r--r-- 1 root root 1984 Jun 27 16:20 ca.crt
-rw-r--r-- 1 root root 3243 Jun 27 16:20 ca.key
-rw-r--r-- 1 root root 5227 Jun 27 16:20 ca.pem
[Jun 27 2024 16:26:21 UTC (QA Tests)] INFO  -- Executing: `docker exec gitaly bash -c 'ls -al /etc/gitlab/ssl'`
[Jun 27 2024 16:26:21 UTC (QA Tests)] DEBUG -- Command output:
total 32
drwxr-xr-x 2 root root 4096 Jun 27 16:20 .
drwxrwxr-x 4 root root 4096 Jun 27 16:21 ..
-rw-r--r-- 1 root root 1911 Jun 27 16:20 gitaly.test.crt
-rw-r--r-- 1 root root 1704 Jun 27 16:20 gitaly.test.csr
-rw-r--r-- 1 root root 3247 Jun 27 16:20 gitaly.test.key
-rw-r--r-- 1 root root 1911 Jun 27 16:20 gitlab.test.crt
-rw-r--r-- 1 root root 1704 Jun 27 16:20 gitlab.test.csr
-rw-r--r-- 1 root root 3243 Jun 27 16:20 gitlab.test.key
Edited by John McDonnell

Merge request reports