[Spike] Replace health-check gem
The health_check
gem does not support Sidekiq 7 due to a new Ruby dependency redis-client
in version 7.
Additionally, this gem also blocks this issue: Remove Flipper references and gem (#4040) because only the health_check
gem uses flipper
at the moment to store state of feature flags for the maintenace_mode
.
We use the health_check
to run our health checks against:
- Zuora
- GitLab.com
- Salesforce
- Mailgun
Additionally, we have enabled built-in checks for:
- Database
- Pending migrations
- Email (which is redundant because we have Mailgun check)
- Cache
- Redis
- Sidekiq-redis
- Queues
Worth mentioning that in the development
environment those check throw the following error:
worker | 2024-04-12T06:49:16.733Z pid=88999 tid=2q2z WARN: {"context":"Job raised exception","job":{"retry":true,"queue":"health_check","class":"HealthCheckCron::CheckGitlabJob","args":[],"jid":"b45e4cf13fa732fbc19db4e5","created_at":1709886720.6729002,"correlation_id":"b7309f54a9eef8311f8b2f98ac797fb5","enqueued_at":1712904556.725954,"error_message":"undefined method `jid=' for #<HealthCheckCron::CheckGitlabJob:0x0000000120c3cfa0 @arguments=[], @job_id=\"e87e0d48-fcec-4192-86d9-43799ef45a23\", @queue_name=#<Proc:0x0000000120c3eb98 /Users/vslobodin/Development/gitlab/customers-gitlab-com/app/jobs/health_check_cron/base_job.rb:41>, @priority=nil, @executions=0, @exception_executions={}, @timezone=\"UTC\">","error_class":"NoMethodError","failed_at":1709886721.038849,"retry_count":20,"retried_at":1712731593.7384288}}
worker | 2024-04-12T06:49:16.733Z pid=88999 tid=2q57 class=HealthCheckCron::CheckGitlabJob jid=dd795c991d55b19c8a5b70d1 INFO: start
worker | 2024-04-12T06:49:16.734Z pid=88999 tid=2q2z WARN: NoMethodError: undefined method `jid=' for #<HealthCheckCron::CheckGitlabJob:0x0000000125677b38 @arguments=[], @job_id="4365601f-d48a-4575-89a8-e3101d3a2012", @queue_name=#<Proc:0x00000001222b19c0 /Users/vslobodin/Development/gitlab/customers-gitlab-com/app/jobs/health_check_cron/base_job.rb:41>, @priority=nil, @executions=0, @exception_executions={}, @timezone="UTC">
worker | 2024-04-12T06:49:16.734Z pid=88999 tid=2q2z WARN: /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:134:in `block (5 levels) in dispatch'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/rails.rb:14:in `block in call'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/execution_wrapper.rb:92:in `wrap'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/reloader.rb:72:in `block in wrap'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/execution_wrapper.rb:92:in `wrap'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/reloader.rb:71:in `wrap'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/rails.rb:13:in `call'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'
worker | /Users/vslobodin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'
Proposal
I think we don't need built-in checks as we have them covered by our infrastructure. Our custom health checks can be implemented as regular Sidekiq jobs. Rails 7.1 has built-in support for the /health
endpoint and we can use it.
Edited by Vitaly Slobodin