After restoring of a backup, gitlab shared runner cannot register anymore. Morover admin/runner GUI results in 500 error. Hence there seems to be no way to get to a new runner token

Update: The issue has been resolved. See this comment for details: #35464 (comment 242397430)

It may be helpful to update the troubleshooting doc: https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/troubleshooting/index.md#included-gitlab-runner-failing-to-register

---

Hello together, I have used the Gitlab helm chart to install gitlab 12.4.1-ee on kubernetes. I am currently trying to backup and restore my installation, unfortunately without success. By backup, which I would like to restore, is from the same gitlab version / installation.

I can follow the documentation from here

https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/backup-restore/restore.md

to restore the database, minio buckets and secrets.Hence, I can follow till section "Restore the runner registration token"

The documentation than refers to https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/troubleshooting/index.md#included-gitlab-runner-failing-to-register

to restore the runner registration token. Unfortunately, the described procedure is not working for me as as the GUI admin/runner page is not showing up, but an 500 error screen instead (hence I cannot read out the new shared runner token). According to the documentation:

Find the new shared runner token located on the admin/runners webpage of your GitLab installation.

As already mentioned, when I go to this page I just get a 500 error page and have found no way to read out the shared runner token. Unfortunately, I have found no description so far how I could get to the token in an alternative way, maybe read it out from postgresql database?

Could someone provide a workaround till this bug is fixed?

Note: I have also found the description here https://docs.gitlab.com/ee/raketasks/backup_restore.html#when-the-secrets-file-is-lost

and tried out the following:

git@gitlab-task-runner-5644756466-7kfhv:/$ gitlab-rails console
WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell.
--------------------------------------------------------------------------------
 GitLab:       12.4.1-ee (8953eae8)
 GitLab Shell: Unknown
 PostgreSQL:   10.10
--------------------------------------------------------------------------------
Loading production environment (Rails 5.2.3)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true
irb(main):002:0> exit
git@gitlab-task-runner-5644756466-7kfhv:/$ 

However, one the one hand I do have my secrets file available, and on the other hand the scatched approach does to work. Would be very great if someone could provide help. I am quite lost on the issue currently :(

For completeness, here is the log output of gitlab-runner, the container that hangs on startup. So it seems clear that the registration token is the problem.

ERROR: Registering runner... forbidden (check registration token) runner=Au8DARBe
PANIC: Failed to register this runner. Perhaps you are having network problems
Registration attempt 25 of 30
Runtime platform arch=amd64 os=linux pid=520 revision=05161b14 version=12.4.1
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...

So I assume, the solution for me would be if I could get to the new registration token for the shared runner in a different way than via UI. By this assumption I am also assuming that the 500 error will go away once the pod is up again?

Thanks for your help.