500 internal error when accessing /settings/integrations
I have upgraded from Gitlab CE 11.11.2 [source] to Gitlab EE 12.1.6 [omnibus] and I have lost my old secrets file. When attempting to access one of my projects [websites/project1] integration pages, [example: https://gitlab_url/websites/project1/-/settings/integrations, it gives a 500 error.
Started GET "/websites/project1/-/settings/integrations" for 10.0.2.5 at 2019-08-20 09:46:15 -0500
Processing by Projects::Settings::IntegrationsController#show as HTML
Parameters: {"namespace_id"=>"websites", "project_id"=>"project1"}
Completed 500 Internal Server Error in 130ms (ActiveRecord: 12.9ms | Elasticsearch: 0.0ms)
ActionView::Template::Error ():
1: %li
2: .row
3: .col-md-8.col-lg-7
4: %strong.light-header= hook.url
5: %div
6: - ProjectHook.triggers.each_value do |event|
7: - if hook.public_send(event)
app/views/projects/settings/integrations/_project_hook.html.haml:4:in `_app_views_projects_settings_integrations__project_hook_html_haml__4601418268038972287_69985572746520'
app/views/projects/hooks/_index.html.haml:20:in `block in _app_views_projects_hooks__index_html_haml__673078787241957713_69985666012520'
app/views/projects/hooks/_index.html.haml:19:in `_app_views_projects_hooks__index_html_haml__673078787241957713_69985666012520'
app/views/projects/settings/integrations/show.html.haml:4:in `_app_views_projects_settings_integrations_show_html_haml___1426220420582464078_69985661964960'
app/controllers/application_controller.rb:117:in `render'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:28:in `set_current_ip_address'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:445:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:439:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
ee/lib/gitlab/jira/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
I tried resetting my system based on https://docs.gitlab.com/debug/gitlab_rails_cheat_sheet.html#bad-decrypt-script
sudo gitlab-rails c
--------------------------------------------------------------------------------
GitLab: 12.1.6-ee (d05ee0a9c12)
GitLab Shell: 9.3.0
PostgreSQL: 9.6.11
--------------------------------------------------------------------------------
Loading production environment (Rails 5.2.3)
irb(main):001:0>
total = 0
bad = []
ProjectImportData.find_each do |data|
begin
total += 1
data.credentials
rescue => e
bad << data
end
end
puts "Bad count: #{bad.count} / #{total}"
bad.each do |repo|
puts Project.find(repo.project_id).full_path
end; bad.count
exit
And I tried this from [ https://docs.gitlab.com/debug/gitlab_rails_cheat_sheet.html#bad-decrypt-script ]
wget -O /tmp/bad-decrypt.rb https://gitlab.com/snippets/1730735/raw
sudo gitlab-rails runner /tmp/bad-decrypt.rb
wget -O /tmp/encrypted-tokens.rb https://gitlab.com/snippets/1876342/raw
gitlab-rails runner /tmp/encrypted-tokens.rb
and I tried
sudo gitlab-rails c
--------------------------------------------------------------------------------
GitLab: 12.1.6-ee (d05ee0a9c12)
GitLab Shell: 9.3.0
PostgreSQL: 9.6.11
--------------------------------------------------------------------------------
Loading production environment (Rails 5.2.3)
Project.find_by_full_path('websites/project1').update(runners_token: nil, runners_token_encrypted:nil)
ApplicationSetting.all.each{|r| r.runners_registration_token}; nil
ApplicationSetting.all.each{|r| r.health_check_access_token}; nil
Ci::Runner.all.each{|r| r.token}; nil
Ci::Runner.all.update_all(token_encrypted: nil)
Ci::Build.all.each{|r| r.token}; nil
DeployToken.all.each{|r| r.project_deploy_tokens}; nil
Group.all.each{|r| r.runners_token}; nil
PersonalAccessToken.all.each{|r| r.token}; nil
Project.all.each{|r| r.runners_token}; nil
User.all.each{|r| r.incoming_email_token}; nil
User.all.each{|r| r.feed_token}; nil
exit
And I tried the following from https://docs.gitlab.com/ee/raketasks/backup_restore.html
sudo gitlab-rails dbconsole
SELECT * FROM public."ci_group_variables";
SELECT * FROM public."ci_variables";
DELETE FROM ci_group_variables;
DELETE FROM ci_variables;
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE ci_runners SET token = null, token_encrypted = null;
\q
and finally, I restarted my Gitlab system
sudo gitlab-ctl restart
and still nothing. It keeps showing a 500 error page. Any idea?
Edited by Boris