fix webhook functionality after gitlab secrets is lost
Problem to solve
Our gitlab secrets file have been lost, I've read When the secrets file is lost and successfully get gitlab-runners to work again.
But as for project webhooks, its pages are reporting 500, here's the relevant logs (collected using k logs -f -l release=gitlab --all-containers=true --max-log-requests=100
):
*** /var/log/gitlab/production.log ***
Started GET "/ein/economist/hooks" for 10.233.13.171 at 2020-05-30 03:19:37 +0000
Processing by Projects::HooksController#index as HTML
Parameters: {"namespace_id"=>"ein", "project_id"=>"economist"}
Completed 500 Internal Server Error in 118ms (ActiveRecord: 7.2ms | Elasticsearch: 0.0ms | Allocations: 52306)
ActionView::Template::Error ():
1: %li
2: .row
3: .col-md-8.col-lg-7
4: %strong.light-header= hook.url
5: %div
6: - hook.class.triggers.each_value do |trigger|
7: - if hook.public_send(trigger)
app/views/shared/web_hooks/_hook.html.haml:4
app/views/shared/web_hooks/_index.html.haml:11
app/views/shared/web_hooks/_index.html.haml:10
app/views/projects/hooks/index.html.haml:14
app/controllers/application_controller.rb:132:in `render'
app/controllers/application_controller.rb:496:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:487:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:481:in `set_locale'
lib/gitlab/error_tracking.rb:48:in `with_context'
app/controllers/application_controller.rb:546:in `sentry_context'
app/controllers/application_controller.rb:474:in `block in set_current_context'
lib/gitlab/application_context.rb:52:in `block in use'
lib/gitlab/application_context.rb:52:in `use'
lib/gitlab/application_context.rb:20:in `with_context'
app/controllers/application_controller.rb:467:in `set_current_context'
lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:56:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/multipart.rb:125:in `call'
lib/gitlab/middleware/read_only/controller.rb:51:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
*** /var/log/gitlab/production_json.log ***
{"method":"GET","path":"/ein/economist/hooks","format":"html","controller":"Projects::HooksController","action":"index","status":500,"time":"2020-05-30T03:19:37.503Z","params":[{"key":"namespace_id","value":"ein"},{"key":"project_id","value":"economist"}],"remote_ip":"10.233.13.171","user_id":2,"username":"timfeirg","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.30 Safari/537.36","queue_duration_s":0.004602,"correlation_id":"6kidEAEck17","meta.user":"timfeirg","meta.project":"ein/economist","meta.root_namespace":"ein","meta.caller_id":"Projects::HooksController#index","redis_calls":6,"redis_duration_s":0.002298,"cpu_s":0.13,"exception.class":"ActionView::Template::Error","exception.message":"","exception.backtrace":["app/views/shared/web_hooks/_hook.html.haml:4","app/views/shared/web_hooks/_index.html.haml:11","app/views/shared/web_hooks/_index.html.haml:10","app/views/projects/hooks/index.html.haml:14","app/controllers/application_controller.rb:132:in `render'","app/controllers/application_controller.rb:496:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:487:in `set_session_storage'","app/controllers/application_controller.rb:481:in `set_locale'","lib/gitlab/error_tracking.rb:48:in `with_context'","app/controllers/application_controller.rb:546:in `sentry_context'","app/controllers/application_controller.rb:474:in `block in set_current_context'","lib/gitlab/application_context.rb:52:in `block in use'","lib/gitlab/application_context.rb:52:in `use'","lib/gitlab/application_context.rb:20:in `with_context'","app/controllers/application_controller.rb:467:in `set_current_context'"],"db_duration_s":0.00724,"view_duration_s":0.0,"duration_s":0.118}
I tried to reset the webhook url inside rails console following gitlab-foss#53763 (comment 116055560), but it just crashes with CipherError:
irb(main):005:0> h = WebHook.find_by(id: 2)
=> #<ProjectHook id: 2, project_id: 14, created_at: "2020-02-13 06:08:13", updated_at: "2020-02-14 02:20:49", type: "ProjectHook", service_id: nil, push_events: true, issues_events: true, merge_requests_events: true, tag_push_events: true, group_id: nil, note_events: true, enable_ssl_verification: true, wiki_page_events: true, pipeline_events: false, confidential_issues_events: true, repository_update_events: false, job_events: true, confidential_note_events: true, push_events_branch_filter: "", encrypted_token: [FILTERED], encrypted_token_iv: nil, encrypted_url: "vxjKnGmVrX6J4PFdIlOw4dhFS9VaMkfhZwGOrh4RUNe9hPUUXX...", encrypted_url_iv: "FPy2Zn3nTf6dc6d2\n", token: nil, url: nil>
irb(main):006:0> h.update(url: "http://foo.bar")
Traceback (most recent call last):
1: from (irb):6
OpenSSL::Cipher::CipherError ()
irb(main):007:0> h.url
Traceback (most recent call last):
2: from (irb):7
1: from (irb):7:in `rescue in irb_binding'
OpenSSL::Cipher::CipherError ()
How can I reset the urls for gitlab webhooks?
Proposal
Add steps to reset webhooks urls (or delete them) in https://docs.gitlab.com/ee/raketasks/backup_restore.html#when-the-secrets-file-is-lost
Or even better, add a procedure to remove all encrypted data from gitlab, so that in the unfortunate event of secret lost, user can proceed without gitlab blowing up.