Cannot open issue page 500 Internal Server Error ee 11

Summary

Couple days ago after we migrated to version 11 the issue pages stopped working. We can't open a page because gitlab return 500 Internal Server Error

Steps to reproduce

  1. Create an issue
  2. Go to the issue's page
  3. This will generate 500 error

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)

What is the current bug behavior?

Displaying 500 error

What is the expected correct behavior?

Display the issue page

Relevant logs and/or screenshots

500

I've found the following in production.log

Started POST "/mygroup/myproject/-/jobs/4/cancel" for 10.20.10.136 at 2019-05-27 15:21:00 +0200
Processing by Projects::JobsController#cancel as HTML
  Parameters: {"authenticity_token"=>"[FILTERED]", "namespace_id"=>"mygroup", "project_id"=>"myproject", "id"=>"4"}
Completed 500 Internal Server Error in 77ms (ActiveRecord: 11.4ms | Elasticsearch: 0.0ms)

OpenSSL::Cipher::CipherError ():

lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'
app/models/concerns/token_authenticatable.rb:40:in `block in add_authentication_token_field'
app/controllers/projects/jobs_controller.rb:100:in `cancel'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:439:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:435:in `set_locale'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
ee/lib/gitlab/jira/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)




System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     go1.11.5 linux/amd64

GitLab information
Version:        11.11.0-ee
Revision:       deb6f84e91f
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.11
URL:            http://gitlab.select.com
HTTP Clone URL: http://gitlab.select.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.select.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        9.1.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git





(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)




Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
5/1 ... yes
5/3 ... yes
5/4 ... yes
7/5 ... yes
5/6 ... yes
5/8 ... yes
5/9 ... yes
5/10 ... yes
11/11 ... yes
5/12 ... yes
5/13 ... yes
5/14 ... yes
5/15 ... yes
5/16 ... yes
5/17 ... yes
5/19 ... yes
5/20 ... yes
5/22 ... yes
5/23 ... yes
5/24 ... yes
5/25 ... yes
5/26 ... yes
5/27 ... yes
5/28 ... yes
5/29 ... yes
5/30 ... yes
5/31 ... yes
5/32 ... yes
5/34 ... yes
5/35 ... yes
5/36 ... yes
5/37 ... yes
5/38 ... yes
5/39 ... yes
5/40 ... yes
5/41 ... yes
5/43 ... yes
5/44 ... yes
5/45 ... yes
5/46 ... yes
5/47 ... yes
5/48 ... yes
5/49 ... yes
5/50 ... yes
5/51 ... yes
19/52 ... yes
5/53 ... yes
19/54 ... yes
19/55 ... yes
5/56 ... yes
5/57 ... yes
5/58 ... yes
5/59 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 20
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Backup your DB and sudo gitlab-rails dbconsole

-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;
Edited by Nolaan