JiraService "OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed"

Dev: https://dev.gitlab.org/gitlab/gitlabhq/issues/2458

https://rpm.newrelic.com/accounts/543063/applications/4968123/traced_errors/6fb7da-4ec66d4e-1f1c-11e5-93a1-f8bc12425d4c

Error message
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed

Stack trace (show Rails)
        /opt/gitlab/embedded/lib/ruby/2.1.0/net/ http.rb: 923:in `connect'
        /opt/gitlab/embedded/lib/ruby/2.1.0/net/ http.rb: 923:in `block in connect'
         /opt/gitlab/embedded/lib/ruby/2.1.0/ timeout.rb:  76:in `timeout'
        /opt/gitlab/embedded/lib/ruby/2.1.0/net/ http.rb: 923:in `connect'
        /opt/gitlab/embedded/lib/ruby/2.1.0/net/ http.rb: 863:in `do_start'
        /opt/gitlab/embedded/lib/ruby/2.1.0/net/ http.rb: 852:in `start'
        /opt/gitlab/embedded/lib/ruby/2.1.0/net/ http.rb:1375:in `request'
…uby/2.1.0/gems/httparty-0.13.3/lib/httparty/ request.rb:  98:in `perform'
…ce/gem/ruby/2.1.0/gems/httparty-0.13.3/lib/ httparty.rb: 539:in `perform_request'
…ce/gem/ruby/2.1.0/gems/httparty-0.13.3/lib/ httparty.rb: 475:in `get'
…tlab-rails/app/models/project_services/ jira_service.rb: 112:in `test_settings'
…tlab-rails/app/models/project_services/ jira_service.rb:  75:in `execute'
…ice/gitlab-rails/app/workers/ project_service_worker.rb:   8:in `perform'
…ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/ processor.rb:  75:in `execute_job'
…ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/ processor.rb:  52:in `block (2 levels) in process'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 127:in `call'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 127:in `block in invoke'
…b-rails/lib/gitlab/sidekiq_middleware/ memory_killer.rb:  17:in `call'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 129:in `block in invoke'
…ails/lib/gitlab/sidekiq_middleware/ arguments_logger.rb:   6:in `call'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 129:in `block in invoke'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 129:in `block in invoke'
…0/gems/sidetiq-0.6.3/lib/sidetiq/middleware/ history.rb:   8:in `call'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 129:in `block in invoke'
…q-3.3.0/lib/sidekiq/middleware/server/ active_record.rb:   6:in `call'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 129:in `block in invoke'
…ekiq-3.3.0/lib/sidekiq/middleware/server/ retry_jobs.rb:  74:in `call'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 129:in `block in invoke'
…sidekiq-3.3.0/lib/sidekiq/middleware/server/ logging.rb:  11:in `block in call'
…m/ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/ logging.rb:  22:in `with_context'
…sidekiq-3.3.0/lib/sidekiq/middleware/server/ logging.rb:   7:in `call'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 129:in `block in invoke'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 132:in `call'
…1.0/gems/sidekiq-3.3.0/lib/sidekiq/middleware/ chain.rb: 132:in `invoke'
…ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/ processor.rb:  51:in `block in process'
…ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/ processor.rb:  98:in `stats'
…ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/ processor.rb:  50:in `process'
…uby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/ calls.rb:  26:in `public_send'
…uby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/ calls.rb:  26:in `dispatch'
…uby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/ calls.rb: 122:in `dispatch'
…ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/ cell.rb:  60:in `block in invoke'
…ruby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/ cell.rb:  71:in `block in task'
…uby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/ actor.rb: 357:in `block in task'
…uby/2.1.0/gems/celluloid-0.16.0/lib/celluloid/ tasks.rb:  57:in `block in initialize'
…ems/celluloid-0.16.0/lib/celluloid/tasks/ task_fiber.rb:  15:in `block in create'

Job

@marin can we pass failing checks on to the user? How would this affect the user / integration and do you think we can improve it?

Marin

@job I think it would be good to catch this exception and handle it gracefully(maybe print the error to the log so GitLab admin can search and fix). I don't think it is good to show this to the end user, might qualify as information leakage.

Jacob

@marin for gitlab.com I (as an admin) don't want to search for some user's SSL error, I can imagine this applies to other installations too. A middle ground might be to allow project services to pass back messages when a test fails, and to use a whitelist for the errors to avoid information leakage.

begin
  test stuff
  return [true, '']
rescue => e
  if ERROR_WHITELIST.include?(e.class.to_s)
    error_message = e.class.to_s
  else
    error_message = 'Unknown error. Ask an administrator to inspect production.log.'
  end
  return [false, error_message]
end

Marin

@jacobvosmaer I can agree that an admin doesn't want to scroll through logs to find the particular error. How do we decide which errors should be whitelisted(one users ok is another users security issue)?

Jacob

@marin use our good judgment?

Marin

use our good judgment? Good luck with that. Having a whitelist almost defeats the purpose of showing the error as a new error will require admin to go into the logs and then contact us to add it to the whitelist and so on.

Jacob

require admin to go into the logs and then contact us to add it to the whitelist and so on I suspect that this feedback loop will stabilize.

This whole thing is just part of having and maintaining project services in my opinion.

a) somebody writes a service and does not understand / know common error conditions
b) we/users/customers observe errors in real deployments
c) we improve the service to handle or report the errors
d) goto b)

cc/ @marin @JobV @jacobvosmaer @dzaporozhets

Edited Oct 12, 2020 by 🤖 GitLab Bot 🤖