pipelines: HTTP/500 after update from 11.8.1 to 11.10.x:

First, sorry for the lengthy post, but things are "a little bit complicated" ...

This is somehow similar to https://gitlab.com/gitlab-org/gitlab-ce/issues/60728

Our story: We've recently updated our omnibus CE installation from to 11.8.1 to 11.10.2 and then, today, to 11.10.4.

After we've made the first update step to 11.10.2, people have started reporting that the pipelines stopped working and production.log showed those exceptions when trying to run a pipeline:

Started POST "/api/v4/jobs/request" for 192.168.1.1 at 2019-05-02 00:35:47 +0200
Processing by Gitlab::RequestForgeryProtection::Controller#index as JSON
  Parameters: {"info"=>{"name"=>"gitlab-ci-multi-runner", "version"=>"9.5.1", "revision"=>"96b34cc", "platform"=>"linux", "architecture"=>"amd64", "executor"=>"docker", "features"=>{"variables"=>"[FILTERED]", "image"=>true, "services"=>true, "features"=>false, "cache"=>false}}, "token"=>"[FILTERED]", "last_update"=>"c4f1bf43fed416d6c267be28c41e1f60", "request_forgery_protection"=>{"info"=>{"name"=>"gitlab-ci-multi-runner", "version"=>"9.5.1", "revision"=>"96b34cc", "platform"=>"linux", "architecture"=>"amd64", "executor"=>"docker", "features"=>{"variables"=>"[FILTERED]", "image"=>true, "services"=>true, "features"=>false, "cache"=>false}}, "token"=>"[FILTERED]", "last_update"=>"c4f1bf43fed416d6c267be28c41e1f60"}}
Can't verify CSRF token authenticity.
This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`
Unlike the logs may suggest, this does not result in an actual 422 response to the user
For API requests, the only effect is that `current_user` will be `nil` for the duration of the request
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms)

OpenSSL::Cipher::CipherError ():
  /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'
  /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
  /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'

I then noticed, that I could not access the runners admin page at /admin/runners (HTTP/500) and production.log showed this:

Started GET "/admin/runners" for 192.168.1.1 at 2019-05-02 11:40:26 +0200
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 257ms (ActiveRecord: 20.3ms)

ActionView::Template::Error ():
    16:   .table-section.section-10
    17:     .table-mobile-header{ role: 'rowheader' }= _('Runner token')
    18:     .table-mobile-content
    19:       = link_to runner.short_sha, admin_runner_path(runner)
    20:
    21:   .table-section.section-20
    22:     .table-mobile-header{ role: 'rowheader' }= _('Description')

lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable.rb:32:in `block in add_authentication_token_field'
app/models/ci/runner.rb:213:in `short_sha'

I then decided to check, if there is an updates CE version available and indeed, 11.10.4 was available. Unfortunately that did not change anything.

As indicated in https://gitlab.com/gitlab-org/gitlab-ce/issues/55596, I've since tried to reset the runners tokens using the rails console:

ApplicationSetting.current.reset_runners_registration_token!

But that didn't change anything.

I then followed https://docs.gitlab.com/ee/raketasks/backup_restore.html#reset-runner-registration-tokens and cleared all project, group, instance and runner tokens.

Now, /admin/runners is responding again, but if I try to open individual CI settings for a project, I get HTTP/500 again and the log says this:

Started GET "/fancy/project/settings/ci_cd" for 94.198.139.2 at 2019-05-02 12:25:43 +0200
Processing by Projects::MergeRequestsController#show as JSON
  Parameters: {"serializer"=>"widget", "namespace_id"=>"fancy", "project_id"=>"fancyness", "id"=>"22"}
Processing by Projects::Settings::CiCdController#show as HTML
  Parameters: {"namespace_id"=>"fancy", "project_id"=>"fancyfancy"}
Completed 500 Internal Server Error in 377ms (ActiveRecord: 42.5ms)
  
ActionView::Template::Error ():
    25:     project_clusters_path(@project),
    26:     class: 'btn btn-info'
    27:   %hr
    28:   = render partial: 'ci/runner/how_to_setup_runner',
    29:            locals: { registration_token: @project.runners_token,
    30:                      type: 'specific',
    31:                      reset_token_url: reset_registration_token_namespace_project_settings_ci_cd_path }
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:45:in `block in add_authentication_token_field'
app/models/project.rb:1550:in `runners_token'

As far as I understand the log entry, the problem still appear to be the runner tokens ...

Any ideas how to proceed? If I can prevent it, I absolutely don't want to reset all CI variables (as indicated in https://docs.gitlab.com/ee/raketasks/backup_restore.html#reset-cicd-variables).

Edited by 🤖 GitLab Bot 🤖