500 errors on Project CI/CD and Kubernetes cluster after restore
Summary
I somehow ended up with a corrupt PostgreSQL DB, and discovered that for some reason my backups had not run for a few months. The most recent backup I had was 11.3.3, but even that wouldn't restore. I ended up having to redeploy my gitlab pod in kubernetes after wiping the iscsi lun it was saving to, then restoring the backup once 11.3.3 was installed again. After installing and restoring, I upgraded 11.3 to 11.10, and all seems to have worked, until I checked on the CI/CD tab and kubernetes tabs. Both are producing 500 errors, and kubernetes should not show any integrations:
I followed both the steps in #52776 (moved), a ticket i opened many months ago, as well as the following: https://stackoverflow.com/questions/54216933/internal-server-error-500-while-accessing-gitlab-admin-runners
root@gitlab:/# gitlab-rails console
-------------------------------------------------------------------------------------
GitLab: 11.5.1 (c90ae59)
GitLab Shell: 8.4.1
postgresql: 9.6.8
-------------------------------------------------------------------------------------
Both Deployment and its :status machine have defined a different default for "status". Use only one or the other for defining defaults to avoid unexpected behaviors.
Loading production environment (Rails 4.2.10)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true
As well as
https://gitlab.com/gitlab-org/gitlab-ce/issues/47922
I used
gitlab-rails console
and thenupdate projects set runners_token_encrypted=NULL where name='<project_name>';
I also tried the solution from this: https://gitlab.com/gitlab-org/gitlab-ce/issues/55596#note_142836563
I did make sure to restore my gitlab-secrets.json, but i have two copies now and not sure which is which.
Steps to reproduce
(How one can reproduce the issue - this is very important)
Example Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behaviour, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)
What is the current bug behavior?
(What actually happens)
What is the expected correct behavior?
It should work
Relevant logs and/or screenshots
Clicking on Kubernetes:
Started GET "/erichardson/py-eagle-mqtt/clusters/1" for 199.167.54.229 at 2019-05-10 21:45:48 +0000
Processing by Projects::ClustersController#show as HTML
Parameters: {"namespace_id"=>"erichardson", "project_id"=>"py-eagle-mqtt", "id"=>"1"}
Completed 500 Internal Server Error in 90ms (ActiveRecord: 9.1ms)
ActionView::Template::Error (undefined method `legacy_abac?' for nil:NilClass):
17: install_knative_path: clusterable.install_applications_cluster_path(@cluster, :knative),
18: update_knative_path: clusterable.update_applications_cluster_path(@cluster, :knative),
19: toggle_status: @cluster.enabled? ? 'true': 'false',
20: has_rbac: has_rbac_enabled?(@cluster) ? 'true': 'false',
21: cluster_type: @cluster.cluster_type,
22: cluster_status: @cluster.status_name,
23: cluster_status_reason: @cluster.status_reason,
app/helpers/clusters_helper.rb:21:in `has_rbac_enabled?'
app/views/clusters/clusters/show.html.haml:20:in `_app_views_clusters_clusters_show_html_haml__2310318684998330055_70292699186220'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:434:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Clicking on CI/CD from project settings:
Started GET "/erichardson/py-eagle-mqtt/settings/ci_cd" for 199.167.54.229 at 2019-05-10 21:47:05 +0000
Processing by Projects::Settings::CiCdController#show as HTML
Parameters: {"namespace_id"=>"erichardson", "project_id"=>"py-eagle-mqtt"}
Completed 500 Internal Server Error in 633ms (ActiveRecord: 26.9ms)
ActionView::Template::Error ():
13: %i.fa.fa-edit.btn
14: - else
15: %span.commit-sha
16: = runner.short_sha
17:
18: .float-right
19: - if @project_runners.include?(runner)
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable.rb:32:in `block in add_authentication_token_field'
app/models/ci/runner.rb:213:in `short_sha'
lib/gitlab/metrics/instrumentation.rb:161:in `block in short_sha'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `short_sha'
app/views/projects/runners/_runner.html.haml:16:in `_app_views_projects_runners__runner_html_haml___2982135119713715737_70292553947280'
app/views/projects/runners/_shared_runners.html.haml:23:in `_app_views_projects_runners__shared_runners_html_haml___1674602912576641194_70292551731120'
app/views/projects/runners/_index.html.haml:10:in `_app_views_projects_runners__index_html_haml__3450216894064759475_70292545950700'
app/views/projects/settings/ci_cd/show.html.haml:42:in `_app_views_projects_settings_ci_cd_show_html_haml___3724848938431750519_70292614533420'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:434:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to the GitLab environment info
root@gitlab:/# gitlab-rake gitlab:env:info
System information System: Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.5 Go Version: unknown
GitLab information Version: 11.10.3 Revision: 3d81a2d2f3f Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.11 URL: https://gitlab.domain.com HTTP Clone URL: https://gitlab.domain.com/some-group/some-project.git SSH Clone URL: ssh://git@gitlab.domain.com:port/some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 9.0.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
root@gitlab:/# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.0.0 ? ... OK (9.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 6/1 ... yes 5/3 ... yes 5/4 ... yes 5/5 ... yes 5/6 ... yes 5/7 ... yes 5/8 ... yes 6/9 ... yes 5/10 ... yes 5/11 ... yes 6/12 ... yes 6/13 ... yes 6/14 ... yes 7/17 ... yes 7/18 ... yes 7/19 ... yes 7/20 ... yes 7/21 ... yes 7/22 ... yes 7/23 ... yes 7/24 ... yes 7/25 ... yes 7/26 ... yes 7/27 ... yes 7/28 ... yes 7/29 ... yes 7/30 ... yes 7/31 ... yes 4/32 ... yes 4/33 ... yes 6/35 ... yes 4/37 ... yes 4/40 ... yes 12/43 ... yes 12/44 ... yes 15/45 ... yes 15/46 ... yes 15/47 ... yes 12/48 ... yes 4/49 ... yes 4/50 ... yes 4/51 ... yes 4/52 ... yes 4/53 ... yes 4/54 ... yes 20/57 ... yes 4/59 ... yes 4/60 ... yes 4/61 ... yes 20/62 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 11
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)