Unable to access runners page.
Summary
After upgrading gitlab-ce from 11.9.0 to 11.9.1 i'm unable to access runners page. CI is not working and runners can't communicate with the Gitlab server.
The web shows "Whoops, something went wrong on our end." and the server replies with a 500 internal server error
Steps to reproduce
- Install gitlab-ce 11.9.0 from debian repository
- Update to gitlab-ce 11.9.1 using apt
What is the current bug behavior?
If you try to access the runners (/admin/runners) page from gitlab, or any other runner related page, the following error is shown on production.log:
Started GET "/admin/runners" for 127.0.0.1 at 2019-03-29 10:11:05 +0100 Processing by Admin::RunnersController#index as HTML Completed 500 Internal Server Error in 205ms (ActiveRecord: 8.0ms) ActionView::Template::Error (): 16: .table-section.section-10 17: .table-mobile-header{ role: 'rowheader' }= _('Runner token') 18: .table-mobile-content 19: = link_to runner.short_sha, admin_runner_path(runner) 20: 21: .table-section.section-20 22: .table-mobile-header{ role: 'rowheader' }= _('Description') lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt' app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token' app/models/concerns/token_authenticatable.rb:30:in `block in add_authentication_token_field' app/models/ci/runner.rb:213:in `short_sha' app/views/admin/runners/_runner.html.haml:19:in `_app_views_admin_runners__runner_html_haml__2520204917365740888_70013341861120' app/views/admin/runners/index.html.haml:138:in `block in _app_views_admin_runners_index_html_haml__780624660334386578_70013349839740' app/views/admin/runners/index.html.haml:137:in `_app_views_admin_runners_index_html_haml__780624660334386578_70013349839740' app/controllers/application_controller.rb:116:in `render' lib/gitlab/i18n.rb:55:in `with_locale' lib/gitlab/i18n.rb:61:in `with_user_locale' app/controllers/application_controller.rb:434:in `set_locale' lib/gitlab/middleware/multipart.rb:103:in `call' lib/gitlab/request_profiler/middleware.rb:16:in `call' lib/gitlab/middleware/go.rb:20:in `call' lib/gitlab/etag_caching/middleware.rb:13:in `call' lib/gitlab/middleware/correlation_id.rb:16:in `block in call' lib/gitlab/correlation_id.rb:15:in `use_id' lib/gitlab/middleware/correlation_id.rb:15:in `call' lib/gitlab/middleware/read_only/controller.rb:40:in `call' lib/gitlab/middleware/read_only.rb:18:in `call' lib/gitlab/middleware/basic_health_check.rb:25:in `call' lib/gitlab/request_context.rb:26:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call' lib/gitlab/middleware/release_env.rb:13:in `call' Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2019-03-29 10:11:10 +0100
The web shows a "500 Whoops, something went wrong on our end."
What is the expected correct behavior?
gitlab should show runners configuration instead of 500 internal server error
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 9.8 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.5 Go Version: unknown
GitLab information Version: 11.9.1 Revision: 86f0b5d Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.aoifes.com HTTP Clone URL: https://gitlab.aoifes.com/some-group/some-project.git SSH Clone URL: git@gitlab.aoifes.com:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 8.7.1 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.7.1 ? ... OK (8.7.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 15/2 ... yes 15/3 ... yes 15/4 ... yes 15/5 ... yes 15/6 ... yes 14/18 ... yes 1/19 ... yes 1/31 ... yes 6/32 ... yes 26/33 ... yes 9/34 ... yes 1/35 ... yes 1/36 ... yes 1/37 ... yes 1/38 ... yes 1/39 ... yes 1/41 ... yes 1/42 ... yes 1/43 ... yes 1/45 ... yes 1/46 ... yes 1/47 ... yes 1/49 ... yes 1/50 ... yes 1/51 ... yes 1/52 ... yes 1/53 ... yes 1/54 ... yes 1/55 ... yes 1/56 ... yes 1/57 ... yes 1/58 ... yes 13/59 ... yes 13/60 ... yes 13/61 ... yes 13/62 ... yes 13/63 ... yes 13/64 ... yes 13/65 ... yes 13/66 ... yes 5/67 ... yes 1/68 ... yes 21/95 ... yes 21/99 ... yes 21/101 ... yes 15/102 ... yes 30/103 ... yes 3/106 ... yes 6/107 ... yes 7/139 ... yes 21/140 ... yes 9/141 ... yes 13/142 ... yes 13/143 ... yes 21/144 ... yes 5/145 ... yes 6/147 ... yes 14/148 ... yes 4/149 ... yes 35/150 ... yes 35/151 ... yes 36/153 ... yes 13/154 ... yes 17/155 ... yes 17/156 ... yes 17/157 ... yes 17/158 ... yes 17/159 ... yes 17/160 ... yes 17/161 ... yes 17/162 ... yes 17/163 ... yes 17/164 ... yes 16/165 ... yes 16/166 ... yes 16/167 ... yes 16/168 ... yes 16/169 ... yes 16/170 ... yes 16/171 ... yes 16/172 ... yes 16/173 ... yes 16/174 ... yes 16/175 ... yes 16/176 ... yes 16/177 ... yes 16/178 ... yes 16/179 ... yes 38/180 ... yes 39/181 ... yes 39/182 ... yes 39/183 ... yes 39/184 ... yes 40/185 ... yes 40/186 ... yes 40/187 ... yes 40/188 ... yes 40/189 ... yes 41/190 ... yes 16/191 ... yes 16/192 ... yes 1/193 ... yes 43/195 ... yes 22/196 ... yes 43/197 ... yes 43/198 ... yes 32/200 ... yes 3/202 ... yes 3/203 ... yes 3/204 ... yes 1/205 ... yes 15/206 ... yes 15/207 ... yes 7/209 ... yes 7/210 ... yes 43/211 ... yes 43/212 ... yes 6/213 ... yes 16/214 ... yes 1/215 ... yes 45/216 ... yes 45/217 ... yes 2/218 ... yes 14/219 ... yes 22/220 ... yes 2/221 ... yes 1/222 ... yes 43/223 ... yes 16/224 ... yes 7/225 ... yes 26/227 ... yes 15/229 ... yes 3/230 ... yes 11/231 ... yes 11/232 ... yes 11/233 ... yes 11/234 ... yes 3/235 ... yes 16/236 ... yes 4/237 ... yes 26/239 ... yes 50/240 ... yes 1/241 ... yes 16/242 ... yes 16/243 ... yes 6/245 ... yes 16/247 ... yes 7/248 ... yes 50/249 ... yes 52/250 ... yes 15/251 ... yes 1/252 ... yes 15/253 ... yes 15/254 ... yes 52/255 ... yes 1/256 ... yes 7/259 ... yes 15/260 ... yes 16/261 ... yes 50/262 ... yes 1/263 ... yes 16/264 ... yes 52/265 ... yes 14/267 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 23
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Tried issuing a ApplicationSetting.current.reset_runners_registration_token!
and deleting ci_variables and ci_group_variables from database, without success.