Skip to content

Update `gitlab_workhorse_options` to include specific `-secretPath` option in `lib/support/init.d`

Joel Lopes Da Silva requested to merge JoeKun/gitlab:joel/gitlab_workhorse_options_include_secretPath_for_init.d_scripts into master

What does this MR do and why?

Update gitlab_workhorse_options to include specific -secretPath option in lib/support/init.d, much like what we already do in lib/support/systemd/gitlab-workhorse.service.

User impact

I was recently trying to setup a GitLab Runner on an Ubuntu machine, and every time my pipeline ran with the Docker executor for the GitLab CI job, it would fail with this error:

Getting source from Git repository
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/some_user/some_project/.git/
remote: Internal server error
fatal: unable to access 'https://gitlab.foo.com/some_user/some_project.git/': The requested URL returned error: 500

Indeed, at the time of the failure, I saw logs like this in /var/log/nginx/access.log:

12.34.56.78 - - [24/Nov/2022:09:43:53 -0800] "GET /some_user/some_project.git/info/refs?service=git-upload-pack HTTP/2.0" 500 22 "-" "gitlab-runner 15.5.1 linux/amd64"

and corresponding logs in gitlab-workhorse.log:

time="2022-11-24T09:43:53-08:00" level=error correlation_id=01GJNBT98GW79A2PB551ATF1TM error="preAuthorizeHandler: do request: secret.JWTTokenString: secret.setBytes: read \"./.gitlab_workhorse_secret\": open ./.gitlab_workhorse_secret: no such file or directory" method=GET uri="/some_user/some_project.git/info/refs?service=git-upload-pack"
gitlab.foo.com  - - [2022/11/24:09:43:53 -0800] "GET /some_user/some_project.git/info/refs?service=git-upload-pack HTTP/1.1" 500 22 "" "gitlab-runner 15.5.1 linux/amd64" 0

Solution

It was pretty clear from those logs that gitlab-workhorse was unable to respond to this request for the git-upload-pack service due to its inability to find the secret file.

I looked for the standard way to supply the path to the secret file, and I found it in lib/support/systemd/gitlab-workhorse.service:

ExecStart=/home/git/gitlab-workhorse/gitlab-workhorse  -listenUmask 0 -listenNetwork unix -listenAddr /home/git/gitlab/tmp/sockets/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080 -authSocket /home/git/gitlab/tmp/sockets/gitlab.socket -documentRoot /home/git/gitlab/public -secretPath /home/git/gitlab/.gitlab_workhorse_secret

The important point here is that gitlab-workhorse needs to be started with the -secretPath /home/git/gitlab/.gitlab_workhorse_secret options.

Sure enough, I didn't have this option in my own startup script, because it was modeled after lib/support/init.d/gitlab, since my GitLab instance is running on FreeBSD, and hence doesn't use systemd.

As soon as I added the same option, and restarted GitLab with its associated services, including gitlab-workhorse, things started working.

Here are corresponding logs from /var/log/nginx/access.log after a successful start of the GitLab CI job:

12.34.56.78 - - [24/Nov/2022:11:23:05 -0800] "GET /some_user/some_project.git/info/refs?service=git-upload-pack HTTP/2.0" 401 279 "-" "gitlab-runner 15.5.1 linux/amd64"
12.34.56.78 - gitlab-ci-token [24/Nov/2022:11:23:05 -0800] "GET /some_user/some_project.git/info/refs?service=git-upload-pack HTTP/2.0" 200 162 "-" "gitlab-runner 15.5.1 linux/amd64"
12.34.56.78 - gitlab-ci-token [24/Nov/2022:11:23:06 -0800] "POST /some_user/some_project.git/git-upload-pack HTTP/2.0" 200 131 "-" "gitlab-runner 15.5.1 linux/amd64"
12.34.56.78 - gitlab-ci-token [24/Nov/2022:11:23:06 -0800] "POST /some_user/some_project.git/git-upload-pack HTTP/2.0" 200 16939 "-" "gitlab-runner 15.5.1 linux/amd64"

and from gitlab-workhorse.log:

gitlab.foo.com  - - [2022/11/24:11:23:05 -0800] "GET /some_user/some_project.git/info/refs?service=git-upload-pack HTTP/1.1" 401 279 "" "gitlab-runner 15.5.1 linux/amd64" 30
gitlab.foo.com  - - [2022/11/24:11:23:05 -0800] "GET /some_user/some_project.git/info/refs?service=git-upload-pack HTTP/1.1" 200 162 "" "gitlab-runner 15.5.1 linux/amd64" 126
gitlab.foo.com  - - [2022/11/24:11:23:06 -0800] "POST /some_user/some_project.git/git-upload-pack HTTP/1.1" 200 131 "" "gitlab-runner 15.5.1 linux/amd64" 226
gitlab.foo.com  - - [2022/11/24:11:23:06 -0800] "POST /some_user/some_project.git/git-upload-pack HTTP/1.1" 200 16939 "" "gitlab-runner 15.5.1 linux/amd64" 441

So with this merge request, I want to record what my problem was, and what my process was to find the solution, which is now part of this very description. Additionally, I want to make it easy to anyone else who is using the standard init.d script to avoid this pitfall, by including this important option for gitlab-workhorse.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports