500 errors on Project CI/CD and Kubernetes cluster after restore

Summary

I somehow ended up with a corrupt PostgreSQL DB, and discovered that for some reason my backups had not run for a few months. The most recent backup I had was 11.3.3, but even that wouldn't restore. I ended up having to redeploy my gitlab pod in kubernetes after wiping the iscsi lun it was saving to, then restoring the backup once 11.3.3 was installed again. After installing and restoring, I upgraded 11.3 to 11.10, and all seems to have worked, until I checked on the CI/CD tab and kubernetes tabs. Both are producing 500 errors, and kubernetes should not show any integrations: image

I followed both the steps in gitlab-ce#52776, a ticket i opened many months ago, as well as the following: https://stackoverflow.com/questions/54216933/internal-server-error-500-while-accessing-gitlab-admin-runners

root@gitlab:/# gitlab-rails console
-------------------------------------------------------------------------------------
 GitLab:       11.5.1 (c90ae59)
 GitLab Shell: 8.4.1
 postgresql:   9.6.8
-------------------------------------------------------------------------------------
Both Deployment and its :status machine have defined a different default for "status". Use only one or the other for defining defaults to avoid unexpected behaviors.
Loading production environment (Rails 4.2.10)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true

As well as

https://gitlab.com/gitlab-org/gitlab-ce/issues/47922

I used gitlab-rails console and then update projects set runners_token_encrypted=NULL where name='<project_name>';

I also tried the solution from this: https://gitlab.com/gitlab-org/gitlab-ce/issues/55596#note_142836563

I did make sure to restore my gitlab-secrets.json, but i have two copies now and not sure which is which.

Steps to reproduce

(How one can reproduce the issue - this is very important)

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behaviour, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)

What is the current bug behavior?

(What actually happens)

What is the expected correct behavior?

It should work

Relevant logs and/or screenshots

Clicking on Kubernetes:

Started GET "/erichardson/py-eagle-mqtt/clusters/1" for 199.167.54.229 at 2019-05-10 21:45:48 +0000
Processing by Projects::ClustersController#show as HTML
  Parameters: {"namespace_id"=>"erichardson", "project_id"=>"py-eagle-mqtt", "id"=>"1"}
Completed 500 Internal Server Error in 90ms (ActiveRecord: 9.1ms)

ActionView::Template::Error (undefined method `legacy_abac?' for nil:NilClass):
    17:   install_knative_path: clusterable.install_applications_cluster_path(@cluster, :knative),
    18:   update_knative_path: clusterable.update_applications_cluster_path(@cluster, :knative),
    19:   toggle_status: @cluster.enabled? ? 'true': 'false',
    20:   has_rbac: has_rbac_enabled?(@cluster) ? 'true': 'false',
    21:   cluster_type: @cluster.cluster_type,
    22:   cluster_status: @cluster.status_name,
    23:   cluster_status_reason: @cluster.status_reason,

app/helpers/clusters_helper.rb:21:in `has_rbac_enabled?'
app/views/clusters/clusters/show.html.haml:20:in `_app_views_clusters_clusters_show_html_haml__2310318684998330055_70292699186220'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:434:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Clicking on CI/CD from project settings:

Started GET "/erichardson/py-eagle-mqtt/settings/ci_cd" for 199.167.54.229 at 2019-05-10 21:47:05 +0000
Processing by Projects::Settings::CiCdController#show as HTML
  Parameters: {"namespace_id"=>"erichardson", "project_id"=>"py-eagle-mqtt"}
Completed 500 Internal Server Error in 633ms (ActiveRecord: 26.9ms)

ActionView::Template::Error ():
    13:           %i.fa.fa-edit.btn
    14:     - else
    15:       %span.commit-sha
    16:         = runner.short_sha
    17:
    18:     .float-right
    19:       - if @project_runners.include?(runner)

lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable.rb:32:in `block in add_authentication_token_field'
app/models/ci/runner.rb:213:in `short_sha'
lib/gitlab/metrics/instrumentation.rb:161:in `block in short_sha'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `short_sha'
app/views/projects/runners/_runner.html.haml:16:in `_app_views_projects_runners__runner_html_haml___2982135119713715737_70292553947280'
app/views/projects/runners/_shared_runners.html.haml:23:in `_app_views_projects_runners__shared_runners_html_haml___1674602912576641194_70292551731120'
app/views/projects/runners/_index.html.haml:10:in `_app_views_projects_runners__index_html_haml__3450216894064759475_70292545950700'
app/views/projects/settings/ci_cd/show.html.haml:42:in `_app_views_projects_settings_ci_cd_show_html_haml___3724848938431750519_70292614533420'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:434:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to the GitLab environment info 
root@gitlab:/# gitlab-rake gitlab:env:info


System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.5.3p105
Gem Version:	2.7.6
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.18.1
Sidekiq Version:5.2.5
Go Version:	unknown


GitLab information
Version:	11.10.3
Revision:	3d81a2d2f3f
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	9.6.11
URL:		https://gitlab.domain.com
HTTP Clone URL:	https://gitlab.domain.com/some-group/some-project.git
SSH Clone URL:	ssh://git@gitlab.domain.com:port/some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	9.0.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
root@gitlab:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.0.0 ? ... OK (9.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
6/1 ... yes
5/3 ... yes
5/4 ... yes
5/5 ... yes
5/6 ... yes
5/7 ... yes
5/8 ... yes
6/9 ... yes
5/10 ... yes
5/11 ... yes
6/12 ... yes
6/13 ... yes
6/14 ... yes
7/17 ... yes
7/18 ... yes
7/19 ... yes
7/20 ... yes
7/21 ... yes
7/22 ... yes
7/23 ... yes
7/24 ... yes
7/25 ... yes
7/26 ... yes
7/27 ... yes
7/28 ... yes
7/29 ... yes
7/30 ... yes
7/31 ... yes
4/32 ... yes
4/33 ... yes
6/35 ... yes
4/37 ... yes
4/40 ... yes
12/43 ... yes
12/44 ... yes
15/45 ... yes
15/46 ... yes
15/47 ... yes
12/48 ... yes
4/49 ... yes
4/50 ... yes
4/51 ... yes
4/52 ... yes
4/53 ... yes
4/54 ... yes
20/57 ... yes
4/59 ... yes
4/60 ... yes
4/61 ... yes
20/62 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 11


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Edited by Evan Richardson