tls-ca-file Doesn't Insert Full SSL Chain

Summary

The runner only injects the root OR intermediate and server certificates into the git CI_SERVER_TLS_CA_FILE. This leads to git failing to pull code for jobs since it can't verify the Gitlab server's SSL chain.

Steps to reproduce

Assign a Let's Encrypt SSL cert to your Gitlab server
Register a Docker container runner

docker run -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner   -v /var/run/docker.sock:/var/run/docker.sock   gitlab/gitlab-runner:alpine

Place the Let's Encrypt intermediate / root certs in a file referenced by tls-ca-file in the runner's config. Note: the order of the certs in in the tls-ca-file doesn't seem to matter. I tried changing them around.
Set the runner's log_level to debug
Watch the runner logs docker logs -f gitlab-runner
Execute a job
Search the logs for entries that begin like:

Executing on /runner-RxZxcC_w-project-1-concurrent-0-predefined-1 the #!/usr/bin/env bash

One of them should contain a line that injects the CA root OR intermediate and server cert into a CI_SERVER_TLS_CA_FILE inside the directory the project will be built in (see logs below).
The root or intermediate cert is missing, so git will fail to pull the code down.

What is the current bug behavior?

The full SSL chain specified in tls-ca-file isn't being injected into the build

What is the expected correct behavior?

The full SSL chain specified in tls-ca-file should be injected into the build

Relevant logs and/or screenshots

ca.crt file contents:

gitlab-runner tmp]# awk -v cmd='openssl x509 -noout -subject -issuer ; echo' '       
    /BEGIN/{close(cmd)};{print | cmd}' < /srv/gitlab-runner/config/certs/ca.crt 
subject= /O=Digital Signature Trust Co./CN=DST Root CA X3
issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3

subject= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3

gitlab-runner tmp]# cat /srv/gitlab-runner/config/certs/ca.crt 
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----

Job output:

Running with gitlab-runner 12.2.0 (a987417a)
  on c04d9b0eca3c RxZxcC_w
Using Docker executor with image ruby:2.5 ...
Pulling docker image ruby:2.5 ...
Using docker image sha256:1b5e4540e7038918c20a64a76f30ccfd33bbb63de28f0c805cce286f9d49c785 for ruby:2.5 ...
Running on runner-RxZxcC_w-project-1-concurrent-0 via a08262ef29c8...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/learning/p1/.git/
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.plovich.net/learning/p1.git/': SSL certificate problem: unable to get issuer certificate
ERROR: Job failed: exit code 1

Docker executor debug logs:

...
mkdir -p "/builds/learning/p1.tmp"
echo -n $\'-----BEGIN CERTIFICATE-----\
MIIFXDCCBESgAwIBAgISAzMFO3TALjTcvIOoBuDi4/xoMA0GCSqGSIb3DQEBCwUA\
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD\
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTA5MTQwMDQyNTBaFw0x\
OTEyMTMwMDQyNTBaMB0xGzAZBgNVBAMTEmdpdGxhYi5wbG92aWNoLm5ldDCCASIw\
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPTpHH2hJA3sbhgotqLSGzqU3vwl\
3G/2eA1sTUrz8s/zsB8ZjVp5y23dWjTwPBtDw0RALfAHluh6xwb6yK/8ubM93sqZ\
NqvMrM3/bSbRdjH3zhJMJxyM2RKr2tMVg83dedaXHa6bi4/ha1jnkcLFrOoWzZd3\
CcVssgRExC07QkqzhmpNEIVwsOP6r9sYpYSwrIgoxpSqcrqVko6gDGj80nyc5c0g\
GjuVPCehkn6+fmC/BcOVUysxI1J2tTvPxou1zEL+Hj48Vz13fYfJ+2aX+NZ5WPeV\
7bKmkavYBcimtgna3J1uxGssOyZgivDDxG2mgCztKtR7RAGL34mjmvRU3B8CAwEA\
AaOCAmcwggJjMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI\
KwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUdT9UzaPpZODycb4j2mzt\
B99bAzYwHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYIKwYBBQUH\
AQEEYzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0c2VuY3J5\
cHQub3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0c2VuY3J5\
cHQub3JnLzAdBgNVHREEFjAUghJnaXRsYWIucGxvdmljaC5uZXQwTAYDVR0gBEUw\
QzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDov\
L2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdgB0\
ftqDMa0zEJEhnM4lT0Jwwr/9XkIgCMY3NXnmEHvMVgAAAW0tb+WiAAAEAwBHMEUC\
IQChLjC4Fi2+/wwaYy5qM2DhBrtg3kG1AHZJ1pr25d6Q9gIgKW7Z179Cu46MP1Wz\
TC9uUBddFZKihEOn8NU0TiXqon8AdgBj8tvN6DvMLM8LcoQnV2szpI1hd4+9daY4\
scdoVEvYjQAAAW0tb+XJAAAEAwBHMEUCIQDBmynvfZO/DblOI004/L4uqBR6RcqV\
Qr8RSuYdK8ewrQIgVE2tOCpixmItc6oobkcLpH7zxcOK6ivAVYXa3iSWjRAwDQYJ\
KoZIhvcNAQELBQADggEBAArEhYuX0FOocxg2e6NrENqAb3sujsHtqI4v63CUhyYg\
BawxwhnoGFk7sFXg/8dhEzYzHdKsb7vL2V8X8JhQZKDdP5jZJx9qAOHfDJJ42U30\
eiYmcClIbxVukssfoCl73ymdfc6P9WM/CFAh1eQXGa0G4k+dQGu6ku5BWTK4Lq+r\
ytHT6dQIIPnJnd08Dy3VuEuzwDup4DAGxCdx1pcDd2QklW1jPee98aufW9x8+AGh\
oFOqbs1mxG4Uo2JF776/Kszt2tyBPs9XoEXTHLH9fDvdTupbJgzgZNleWLCj14QW\
plXSbZib6K2WbN0eo4qnJP2W+YJRzGQZQMuYp9cBx7E=\
-----END CERTIFICATE-----\
-----BEGIN CERTIFICATE-----\
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/\
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT\
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow\
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT\
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC\
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF\
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8\
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0\
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA\
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj\
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T\
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG\
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv\
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k\
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw\
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC\
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz\
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu\
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF\
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo\
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/\
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu\
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG\
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6\
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==\
-----END CERTIFICATE-----\
\' > "/builds/learning/p1.tmp/CI_SERVER_TLS_CA_FILE"
export CI_SERVER_TLS_CA_FILE="/builds/learning/p1.tmp/CI_SERVER_TLS_CA_FILE"
...

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information System: Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown

GitLab information Version: 12.2.5 Revision: 09f8edbc29a Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://gitlab.plovich.net HTTP Clone URL: https://gitlab.plovich.net/some-group/some-project.git SSH Clone URL: git@gitlab.plovich.net:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers:

GitLab Shell Version: 9.3.0 Repository storage paths:

default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

GitLab Shell Version: 9.3.0 Repository storage paths:

default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git [root@gitlab ~]# sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/1 ... yes 3/2 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 2

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

Use a self-signed cert