Cannot clone submodules: "SSL certificate problem: unable to get local issuer certificate"

Summary

Attempting clone a submodule from a private GitLab server (same server as superproject) does not work if the HTTPS certificate is signed by a custom CA.

Superproject and submodule are hosted on the same server
OS: Windows server 2012 R2
gitlab-runner version: 11.1.0
git version: 2.7.2.windows.1
Executor: Shell

CI Job Log

Here's the log without any debug information.

Running with gitlab-runner 11.1.0 (081978aa)
  on win-build deadbeef
Using Shell executor...
Running on WIN-BUILD...
Cloning repository...
Cloning into 'C:/Users/gitlab-runner/builds/deadbeef/0/somegroup/someproj'...
Checking out 12345678 as v0.1.1...
Updating/initializing submodules...
Submodule 'some-lib' (https://gitlab-ci-token:xxxxxxxxxxxxxxxx@gitlab.example.com/somegroup/some-lib.git) registered for path 'some-lib'
Cloning into 'some-lib'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxx@gitlab.example.com/somegroup/some-lib.git/': SSL certificate problem: unable to get local issuer certificate
fatal: clone of 'https://gitlab-ci-token:xxxxxxxxxxxxxxxx@gitlab.example.com/somegroup/some-lib.git/' into submodule path 'some-lib' failed
ERROR: Job failed: exit status 128

Debugging Details

I mentioned this there a while back, but didn't debug it quite as far as I have now.

The problem may actually be with Git (2.7.2). Git seems to ignore sslCAInfo for submodules, even those at the same URL as the super-project.

Recall that since !687 (merged), gitlab-runner configures the repository .git/config like this:

[http "https://gitlab.exmaple.com"]
    sslCAInfo = C:\\Users\\gitlab-runner\\builds\\deadbeef\\0\\somegroup\\someproj.git\\CA_SERVER_TLS_CA_FILE

In this comment, @nolith showed an excerpt of this job (which has GIT_CURL_VERBOSE=1) while testing !687 (merged), which sets sslCAInfo for just the Git host. The key part is that CAfile: /nolith/test-runner-2148.tmp/CI_SERVER_TLS_CA_FILE when fetching the super-project (on gitlab.com) and CAfile: /etc/ssl/certs/ca-certificates.crt when fetching a submodule on github.com. That is as expected.

My problem is that I see CAfile: ...tmp\CI_SERVER_TLS_CA_FILE when fetching the super-project (on gitlab.example.com) but then see CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt when fetching the submodule from the same server.

I went as far as configuring a local test repository (on the runner box), which set the sslCAInfo to the same path, and in the same way as the .git/config from the failing project. E.g.

[fetch]
    recurseSubmodules = false
[http "https://gitlab.exmaple.com"]
    sslCAInfo = C:\\Users\\gitlab-runner\\builds\\deadbeef\\0\\somegroup\\someproj.git\\CA_SERVER_TLS_CA_FILE
[core]
    ...
[remote "origin"]
    url = https://jreinhart:<private-access-token>@gitlab.example.com/somegroup/someproj.git
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
    remote = origin
    merge = refs/heads/master
[submodule "some-lib"]
    url = https://jreinhart:<private-access-token>@gitlab.example.com/somegroup/some-lib.git

I was able to successfully issue a git fetch. However, git submodule update failed with SSL certificate problem: unable to get local issuer certificate. After setting GIT_CURL_VERBOSE=1, I would see (abbreviated):

C:\Users\jreinhart\testrepo>git fetch
...
* Connected to gitlab.example.com (x.x.x.x) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:\Users\gitlab-runner\builds\deadbeef\0\somegroup\someproj.git\CA_SERVER_TLS_CA_FILE
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  <certificate details here>
*        SSL certificate verify ok.
...

C:\Users\jreinhart\testrepo>git submodule update
...
* Connected to gitlab.example.com (x.x.x.x) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* SSL certificate problem: unable to get local issuer certificate
...

So it appears that this may be a problem with Git itself. Or, gitlab-runner's expectation of how sslCAInfo works is incorrect.

Related Issues

#2868 (closed)