Skip to content

Fix unintialized constant exception handler error

Stan Hu requested to merge sh-fix-omnibus-helper-constant-error into master

What does this MR do?

When gitlab-ctl reconfigure hits an HTTP error while trying to download local cookbooks, the application runs into an uninitialized constant GitLabHandler::Exception::OmnibusHelper error:

ERROR: Running exception handlers
ERROR: Report handler GitLabHandler::Exception raised #<NameError: uninitialized constant GitLabHandler::Exception::OmnibusHelper>
ERROR: /opt/gitlab/embedded/cookbooks/package/libraries/handlers/gitlab.rb:30:in `report'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:296:in `run_report_unsafe'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:284:in `run_report_safely'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:155:in `block in run_exception_handlers'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:153:in `each'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:153:in `run_exception_handlers'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:164:in `block in <class:Handler>'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:443:in `block in run_failed'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:442:in `each'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:442:in `run_failed'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:306:in `rescue in run'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:299:in `run'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
ERROR: /opt/gitlab/embedded/bin/cinc-client:25:in `load'
ERROR: /opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
ERROR: Exception handlers complete

Related issues

This was seen in gitlab-com/gl-infra/production#16897 (closed) and raised as a side issue in #8245 (closed).

How to validate locally

The easiest way is to hack the Chef instance. Edit /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/util/threaded_job_queue.rb and add an error:

      def process(concurrency = 10)
        workers = (1..concurrency).map do
          Thread.new do
            raise 'error!'
            loop do
              fn = @queue.pop
              fn.arity == 1 ? fn.call(@lock) : fn.call
            end
          end
        end

Diff:

46a47
>           raise 'error!'

Likely you can reproduce this if you run gitlab-ctl reconfigure while apt-get install gitlab-ee is running.

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • MR title and description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • Latest Merge Result pipeline is green.
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

  • The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
  • If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
  • If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes.
  • Documentation created/updated.
  • Tests added.
  • Integration tests added to GitLab QA.
  • Equivalent MR/issue for the GitLab Chart opened.
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.
Edited by João Alexandre Cunha

Merge request reports