Skip to content

Self signed certificate and git lfs results in x509

Summary

Sounds like gitlab-runner does not work by pulling the lfs objects under a self signed certificate. Happened after upgrading my distribution (buster to bullseye) which by the same time upgrade gitlab and gitlab-runner under latest versions.

Before that, it worked fine. please don't come with an answer such as git config ... ssl.verify false...

Notes:

  • i've spend 2 hours looking for a solution, i found none to solve it
  • git clone from runner and local computer works fine with the self signed certificate
  • it's a shell executor

Steps to reproduce

Upgrade gitlab + gitlab-runner on latest versions (14.3) and debian from buster to bullseye

Actual behavior

Runner from the web-ui:

Running with gitlab-runner 14.3.2 (e0218c92)
  on description 5Vip5cCL
Preparing the "shell" executor 00:00
Using Shell executor...
Preparing environment 00:00
Running on xxx...
Getting source from Git repository 00:03
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /home/gitlab-runner/builds/5Vip5cCL/0/thesis/yyy/.git/
Checking out 6f8049b9 as podman...
batch response: Post "https://gitlab-ci-token:***@gitlab.parmentier.io/thesis/yyy.git/info/lfs/objects/batch": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
error: failed to fetch some objects from 'https://gitlab-ci-token:[MASKED]@gitlab.parmentier.io/thesis/yyy.git/info/lfs'
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit status 1

CLI on the runner:

gitlab-runner@xxx:~/builds/5Vip5cCL/0/thesis/yyy$ git lfs pull
batch response: Post "https://gitlab-ci-token:***@gitlab.parmentier.io/thesis/yyy.git/info/lfs/objects/batch": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0                               
error: failed to fetch some objects from 'https://gitlab-ci-token:zzz@gitlab.parmentier.io/thesis/yyy.git/info/lfs'

Environment description

Everything is self hosted.

root@gitlab:/etc/gitlab# gitlab-rake gitlab:env:info

System information
System:         Debian 11
Current User:   git
Using RVM:      no
Ruby Version:   2.7.4p191
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.14
Git Version:    2.33.0.
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        14.3.0
Revision:       ceec8accb09
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://gitlab.parmentier.io
HTTP Clone URL: https://gitlab.parmentier.io/some-group/some-project.git
SSH Clone URL:  ssh://git@gitlab.parmentier.io:24/some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        13.21.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Used GitLab Runner version

gitlab-runner@xxx:~/builds/5Vip5cCL/0/thesis/yyy$ gitlab-runner --version
Version:      14.3.2
Git revision: e0218c92
Git branch:   14-3-stable
GO version:   go1.13.8
Built:        2021-09-30T16:10:52+0000
OS/Arch:      linux/amd64

Possible fixes

Very ugly, I ran on my local computer export GODEBUG=x509ignoreCN=0, then git lfs pull worked. Don't want that fix on my self-hosted prod gitlab-runner.

Others remarks

  1. I found this information

Git LFS relies on Go's crypto/x509 package to find certs, and extends it with support for some of Git's CA config values, specifically http.sslCAInfo/GIT_SSL_CAINFO and http.sslCAPath/GIT_SSL_CAPATH

source

  1. It looks like gitlab-runner handle this through CI_SERVER_TLS_CA_FILE 
root@xxx:/home/gitlab-runner/builds/5Vip5cCL/0/thesis/yyy# cat .git/config 
[fetch]
        recurseSubmodules = false
[http "https://gitlab.parmentier.io"]
        sslCAInfo = /home/gitlab-runner/builds/5Vip5cCL/0/thesis/mary-morstan.tmp/CI_SERVER_TLS_CA_FILE
  1. However the file does not exist ?!
root@xxxx:/home/gitlab-runner/builds/5Vip5cCL/0/thesis/yyyy# ls /home/gitlab-runner/builds/5Vip5cCL/0/thesis/yyyy.tmp/CI_SERVER_TLS_CA_FILE
ls: cannot access '/home/gitlab-runner/builds/5Vip5cCL/0/thesis/yyy.tmp/CI_SERVER_TLS_CA_FILE': No such file or directory
Edited by Laurent Parmentier