Gitlab-Runner logs error 403 after upgrading GitLab FOSS 13.2.8 to 13.3.4
Summary
GitLab Runner cannot connect after upgrading GitLab FOSS from 13.2.8 to 13.3.4
What is the current bug behavior?
After upgrading from release 13.2.x to 13.3.x the shared GitLab-Runner 13.3.0 on another system is no longer able to pick up jobs. The error code in the log is 403.
The following observation might give a hint to the nature of the problem: When opening the page /admin/runners in the browser for the first time after the upgrade the URL of the configuration page for the runner (/admin/runners/3) will appear in the "Runner token" column. After pausing and resuming the runner a wrong token shows up here.
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`) System information System: Current User: gitlab Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 3.0.3 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 6.0.4 Git Version: 2.28.0 Sidekiq Version:5.2.9 Go Version: go1.13.15 linux/amd64 GitLab information Version: 13.3.4 Revision: aebc07f48e8 Directory: /home/gitlab/gitlab DB Adapter: PostgreSQL DB Version: 12.4 URL: https://i30git.itec.kit.edu HTTP Clone URL: https://i30git.itec.kit.edu/some-group/some-project.git SSH Clone URL: gitlab@i30git.itec.kit.edu:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 13.6.0 Repository storage paths: - default: /home/gitlab/repositories GitLab Shell path: /home/gitlab/gitlab-shell Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
diff -U 1 lib/support/init.d/gitlab /etc/init.d/gitlab
--- lib/support/init.d/gitlab 2020-06-26 14:30:59.066722159 +0200
+++ /etc/init.d/gitlab 2020-09-01 18:37:22.733486423 +0200
@@ -8,3 +8,3 @@
# Provides: gitlab
-# Required-Start: $local_fs $remote_fs $network $syslog redis-server
+# Required-Start: $local_fs $remote_fs $network $syslog redis@gitlab
# Required-Stop: $local_fs $remote_fs $network $syslog
diff -U 1 lib/support/init.d/gitlab.default.example /etc/default/gitlab
--- lib/support/init.d/gitlab.default.example 2020-05-23 14:06:06.241744802 +0200
+++ /etc/default/gitlab 2020-09-04 18:11:01.165065528 +0200
@@ -13 +13 @@
# The default is "git".
-app_user="git"
+app_user="gitlab"
@@ -50 +50,2 @@
-gitlab_workhorse_options="-listenUmask 0 -listenNetwork unix -listenAddr $socket_path/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080 -authSocket $socket_path/gitlab.socket -documentRoot $app_root/public"
+# listen on port when using Apache instead of Nginx
+gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp4 -listenAddr localhost:8181 -authBackend http://127.0.0.1:8080 -authSocket $socket_path/gitlab.socket -documentRoot $app_root/public"
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.6.0 ? ... OK (13.6.0) Running /home/gitlab/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... yes Init script up-to-date? ... no Try fixing it: Re-download the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Projects have namespace: ... 10/3 ... yes 10/4 ... yes 15/5 ... yes 16/6 ... yes 16/7 ... yes 10/8 ... yes 13/13 ... yes 22/14 ... yes 23/15 ... yes 23/16 ... yes 23/17 ... yes 23/18 ... yes 22/19 ... yes 23/20 ... yes 23/21 ... yes 22/22 ... yes 22/23 ... yes 24/24 ... yes 23/25 ... yes 52/27 ... yes 23/29 ... yes 15/30 ... yes 15/31 ... yes 13/32 ... yes 15/35 ... yes 22/36 ... yes 10/37 ... yes 21/38 ... yes 21/40 ... yes 21/42 ... yes 12/43 ... yes 21/44 ... yes 52/45 ... yes 23/46 ... yes 52/47 ... yes 52/48 ... yes 52/49 ... yes 52/50 ... yes 52/51 ... yes 52/52 ... yes 52/53 ... yes 52/55 ... yes 10/56 ... yes 23/57 ... yes 15/61 ... yes 52/66 ... yes 22/68 ... yes 10/69 ... yes 21/71 ... yes 123/73 ... yes 22/75 ... yes 10/77 ... yes 21/78 ... yes 15/79 ... yes 52/81 ... yes 52/82 ... yes 16/83 ... yes 16/84 ... yes 16/86 ... yes 16/87 ... yes 16/88 ... yes 15/96 ... yes 13/100 ... yes 51/102 ... yes 51/103 ... yes 51/104 ... yes 52/105 ... yes 23/106 ... yes 15/107 ... yes 15/108 ... yes 51/109 ... yes 13/113 ... yes 15/116 ... yes 15/118 ... yes 15/119 ... yes 15/120 ... yes 15/121 ... yes 15/122 ... yes 56/125 ... yes 52/126 ... yes 58/128 ... yes 21/129 ... yes 52/130 ... yes 52/134 ... yes 56/135 ... yes 15/148 ... yes 12/154 ... yes 52/159 ... yes 15/160 ... yes 52/161 ... yes 13/164 ... yes 58/165 ... yes 15/166 ... yes 21/167 ... yes 21/168 ... yes 52/170 ... yes 56/171 ... yes 56/173 ... yes 52/174 ... yes 52/181 ... yes 52/182 ... yes 52/183 ... yes 56/184 ... yes 15/187 ... yes 56/188 ... yes 22/189 ... yes 15/192 ... yes 56/193 ... yes 22/194 ... yes 22/195 ... yes 56/197 ... yes 15/198 ... yes 56/200 ... yes 15/201 ... yes 52/202 ... yes 12/203 ... yes 15/205 ... yes 22/207 ... yes 13/209 ... yes 52/211 ... yes 13/216 ... yes 13/217 ... yes 88/220 ... yes 22/222 ... yes 88/224 ... yes 56/226 ... yes 56/227 ... yes 19/228 ... yes 88/229 ... yes 52/230 ... yes 52/231 ... yes 13/232 ... yes 13/233 ... yes 22/234 ... yes 56/235 ... yes 13/241 ... yes 116/242 ... yes 52/246 ... yes 52/248 ... yes 56/250 ... yes 15/252 ... yes 13/254 ... yes 56/255 ... yes 13/257 ... yes 13/258 ... yes 13/261 ... yes 118/264 ... yes 116/265 ... yes 116/266 ... yes 13/267 ... yes 23/269 ... yes 13/272 ... yes 13/274 ... yes 116/275 ... yes 13/277 ... yes 56/283 ... yes 23/285 ... yes 56/286 ... yes 13/287 ... yes 23/292 ... yes 122/298 ... yes 122/301 ... yes 122/302 ... yes 122/303 ... yes 116/304 ... yes 130/305 ... yes 116/306 ... yes 52/308 ... yes 23/309 ... yes 116/310 ... yes 130/311 ... yes 122/312 ... yes 122/313 ... yes 130/315 ... yes 116/316 ... yes 116/317 ... yes 130/318 ... yes 130/319 ... yes 130/320 ... yes 116/321 ... yes 116/322 ... yes 116/323 ... yes 116/324 ... yes 88/325 ... yes 88/326 ... yes 15/327 ... yes 134/330 ... yes 134/331 ... yes 134/332 ... yes 134/333 ... yes 116/334 ... yes 130/337 ... yes 116/338 ... yes 116/339 ... yes 22/340 ... yes 22/341 ... yes 116/342 ... yes 116/343 ... yes 23/344 ... yes 22/345 ... yes 88/347 ... yes 116/348 ... yes 116/349 ... yes 116/350 ... yes 12/352 ... yes 23/353 ... yes 116/355 ... yes 116/357 ... yes 116/358 ... yes 116/360 ... yes 116/361 ... yes 116/362 ... yes 116/363 ... yes 116/364 ... yes 130/365 ... yes 22/367 ... yes 155/368 ... yes 155/369 ... yes 140/370 ... yes 122/371 ... yes 12/372 ... yes 12/373 ... yes 130/374 ... yes 130/375 ... yes 140/376 ... yes 140/377 ... yes 140/378 ... yes 116/380 ... yes 140/381 ... yes 143/382 ... yes 142/383 ... yes 139/384 ... yes 122/385 ... yes 23/386 ... yes 23/387 ... yes 116/392 ... yes 148/393 ... yes 146/394 ... yes 23/395 ... yes 116/396 ... yes 122/397 ... yes 147/399 ... yes 116/400 ... yes 12/402 ... yes 155/403 ... yes 146/404 ... yes 136/405 ... yes 136/406 ... yes 150/407 ... yes 122/408 ... yes 116/409 ... yes 152/410 ... yes 152/411 ... yes 150/412 ... yes 116/414 ... yes 23/415 ... yes 116/416 ... yes 142/417 ... yes 139/418 ... yes 140/419 ... yes 153/420 ... yes 116/422 ... yes 151/423 ... yes 140/424 ... yes 22/425 ... yes 140/426 ... yes 140/427 ... yes 140/428 ... yes 147/429 ... yes 122/431 ... yes 140/432 ... yes 157/433 ... yes 154/434 ... yes 116/435 ... yes 150/436 ... yes 140/437 ... yes 116/438 ... yes 154/439 ... yes 150/440 ... yes 157/441 ... yes 12/442 ... yes 150/443 ... yes 23/444 ... yes 116/445 ... yes 116/446 ... yes 116/447 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.24.0 ? ... yes (2.28.0) Git user has default SSH configuration? ... yes Active users: ... 25 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished