Jira Issues Integration: Error 500 when Jira host is unreachable
Summary
Problem: Setting up a jira integration with an unreachable URL results in a 500 level error when attempting to view the Jira Issues page: .../-/integrations/jira/issues
. Because it is a 500 level error this means it can contribute to SLIs for our error rate SLOs on both GitLab.com and GitLab dedicated tenants.
Impact
With sufficiently low ratio of other non-error traffic this error pages on call engineers needlessly. This error should be a different class of error (such as a 400 level error) rather than a server error to avoid showing up in SLO dashboards and causing pages that cannot be handled by the teams being notified.
Example of an issue where the GitLab dedicated team was paged on a new tenant with low traffic as they attempt to setup their instance: https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/64
Additionally it does not give the user feedback that their instance URL is misconfigured only showing the following page:
In the network requests you can see a background request that results in a 500 error and in the logs it also prints a 500 error that actually does include the information that the URL is invalid (example):
"exception.class": "Gitlab::HTTP::BlockedUrlError",
"exception.message": "URL is blocked: Host cannot be resolved or invalid",
"exception.backtrace": [
"lib/gitlab/http_connection_adapter.rb:58:in `rescue in validate_url_with_proxy!'",
"lib/gitlab/http_connection_adapter.rb:50:in `validate_url_with_proxy!'",
"lib/gitlab/http_connection_adapter.rb:27:in `connection'",
"lib/gitlab/http.rb:65:in `perform_request'",
"lib/gitlab/jira/http_client.rb:38:in `public_send'",
"lib/gitlab/jira/http_client.rb:38:in `make_request'",
"lib/gitlab/jira/http_client.rb:13:in `request'",
"app/services/jira/requests/base.rb:53:in `request'",
"app/services/jira/requests/base.rb:27:in `execute'",
"ee/app/finders/projects/integrations/jira/issues_finder.rb:48:in `fetch_issues'",
"ee/app/finders/projects/integrations/jira/issues_finder.rb:36:in `execute'",
"ee/app/controllers/projects/integrations/jira/issues_controller.rb:56:in `issues_json'",
"ee/app/controllers/projects/integrations/jira/issues_controller.rb:30:in `block (2 levels) in index'",
"ee/app/controllers/projects/integrations/jira/issues_controller.rb:27:in `index'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
"ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'",
"app/controllers/application_controller.rb:500:in `set_current_admin'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:491:in `set_session_storage'",
"lib/gitlab/i18n.rb:107:in `with_locale'",
"lib/gitlab/i18n.rb:113:in `with_user_locale'",
"app/controllers/application_controller.rb:482:in `set_locale'",
"app/controllers/application_controller.rb:475:in `set_current_context'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/jira/middleware.rb:19:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:37:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:15:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:44:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
"lib/gitlab/middleware/release_env.rb:13:in `call'"
],
"exception.cause_class": "Gitlab::UrlBlocker::BlockedUrlError",
Recommendation
I have 2 levels of recommendation:
- Simply ensure this error returns a 400 level error in the same manner as it currently returns a 500
- The larger fix would be to surface the actual problem in these cases so that the users can be prompted to fix the bad URL or at least notified that GitLab could not reach the host. That would take some UX and I presume a lot more changes.
Either option would solve the infrastructure problem of paging folks and needlessly contributing to SLO violations.
Verification
This error can be easily reproduced on any GitLab installation (including .com) simply by the following steps:
- On any project (with sufficient license) navigate to
Settings -> Integrations -> Jira
- Enter required details but ensure that
Web URL
is an unreachable host. Either one that does not exist or that is unreachable due to firewalls or other reasons. Save this configuration. - Navigate to Plan -> Jira Issues in the project
- At this point you will see:
- If then you check the network requests in your browser's developer tools you will see a background request to
issues.json
that will return a 500 error instead of any valid JSON.
It should also be possible to search the logs for this error but unfortunately I lack the knowledge of the production infrastructure to know how to find this type of error. It should be sufficient just to ensure that the API does not return a 500 error though to verify this issue is fixed.