Skip to content

Rack::Timeout::RequestTimeoutException on particular pipeline pages

Summary

A large GitLab Ultimate customer on ~SaaS experiences 500 errors for particular pipeline pages and generally "high load times on other[s]" in that project. Self-managed customers are affected as well, with reports from 16.5.1, 16.6.1 and 16.6.2.

Steps to reproduce

  1. Load pipeline page mentioned in this internal support ticket
  2. Check our logs for the reported correlation ID, or for json.exception.class: "Rack::Timeout::RequestTimeoutException" and json.path: "…their…project…".

Example Project

What is the current bug behavior?

500 error page & our Kibana logs point to version_in_default_licenses_range?@ee/app/models/package_metadata/package.rb.

Full json.exception.backtrace 

\\\\\\\[ ee/app/models/package_metadata/package.rb:96:in \\\\\\\`version_in_default_licenses_range?', ee/app/models/package_metadata/package.rb:84:in \\\\\\\`default_license_ids', ee/app/models/package_metadata/package.rb:34:in \\\\\\\`license_ids_for', ee/lib/gitlab/license_scanning/package_licenses.rb:48:in \\\\\\\`block (4 levels) in fetch', ee/lib/gitlab/license_scanning/package_licenses.rb:47:in \\\\\\\`each', ee/lib/gitlab/license_scanning/package_licenses.rb:47:in \\\\\\\`block (3 levels) in fetch', ee/lib/gitlab/license_scanning/package_licenses.rb:46:in \\\\\\\`block (2 levels) in fetch', ee/lib/gitlab/license_scanning/package_licenses.rb:43:in \\\\\\\`each_slice', ee/lib/gitlab/license_scanning/package_licenses.rb:43:in \\\\\\\`each', ee/lib/gitlab/license_scanning/package_licenses.rb:43:in \\\\\\\`block in fetch', lib/gitlab/database/load_balancing/session.rb:72:in \\\\\\\`use_replicas_for_read_queries', ee/lib/gitlab/license_scanning/package_licenses.rb:151:in \\\\\\\`use_replica_if_available', ee/lib/gitlab/license_scanning/package_licenses.rb:31:in \\\\\\\`fetch', ee/lib/gitlab/license_scanning/sbom_scanner.rb:16:in \\\\\\\`report', ee/app/helpers/ee/projects/pipeline_helper.rb:36:in \\\\\\\`license_scan_count', ee/app/helpers/ee/projects/pipeline_helper.rb:20:in \\\\\\\`js_pipeline_tabs_data', app/views/projects/pipelines/show.html.haml:31, app/controllers/application_controller.rb:134:in \\\\\\\`render', app/controllers/projects/pipelines_controller.rb:245:in \\\\\\\`block (2 levels) in render_show', app/controllers/projects/pipelines_controller.rb:243:in \\\\\\\`render_show', app/controllers/projects/pipelines_controller.rb:117:in \\\\\\\`block (2 levels) in show', app/controllers/projects/pipelines_controller.rb:116:in \\\\\\\`show', app/controllers/application_controller.rb:519:in \\\\\\\`block in allow_gitaly_ref_name_caching', lib/gitlab/gitaly_client.rb:472:in \\\\\\\`allow_ref_name_caching', app/controllers/application_controller.rb:518:in \\\\\\\`allow_gitaly_ref_name_caching', ee/lib/gitlab/ip_address_state.rb:10:in \\\\\\\`with', ee/app/controllers/ee/application_controller.rb:45:in \\\\\\\`set_current_ip_address', app/controllers/application_controller.rb:470:in \\\\\\\`set_current_admin', lib/gitlab/session.rb:11:in \\\\\\\`with_session', app/controllers/application_controller.rb:461:in \\\\\\\`set_session_storage', lib/gitlab/i18n.rb:114:in \\\\\\\`with_locale', lib/gitlab/i18n.rb:120:in \\\\\\\`with_user_locale', app/controllers/application_controller.rb:452:in \\\\\\\`set_locale', app/controllers/application_controller.rb:445:in \\\\\\\`set_current_context', ee/lib/omni_auth/strategies/group_saml.rb:41:in \\\\\\\`other_phase', lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in \\\\\\\`call', lib/gitlab/middleware/memory_report.rb:13:in \\\\\\\`call', lib/gitlab/middleware/speedscope.rb:13:in \\\\\\\`call', lib/gitlab/database/load_balancing/rack_middleware.rb:23:in \\\\\\\`call', lib/gitlab/middleware/rails_queue_duration.rb:33:in \\\\\\\`call', lib/gitlab/etag_caching/middleware.rb:21:in \\\\\\\`call', lib/gitlab/metrics/rack_middleware.rb:16:in \\\\\\\`block in call', lib/gitlab/metrics/web_transaction.rb:46:in \\\\\\\`run', lib/gitlab/metrics/rack_middleware.rb:16:in \\\\\\\`call', lib/gitlab/middleware/go.rb:20:in \\\\\\\`call', lib/gitlab/middleware/query_analyzer.rb:11:in \\\\\\\`block in call', lib/gitlab/database/query_analyzer.rb:37:in \\\\\\\`within', lib/gitlab/middleware/query_analyzer.rb:11:in \\\\\\\`call', lib/gitlab/middleware/multipart.rb:173:in \\\\\\\`call', lib/gitlab/middleware/read_only/controller.rb:50:in \\\\\\\`call', lib/gitlab/middleware/read_only.rb:18:in \\\\\\\`call', lib/gitlab/middleware/same_site_cookies.rb:27:in \\\\\\\`call', lib/gitlab/middleware/path_traversal_check.rb:35:in \\\\\\\`call', lib/gitlab/middleware/handle_malformed_strings.rb:21:in \\\\\\\`call', lib/gitlab/middleware/basic_health_check.rb:25:in \\\\\\\`call', lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in \\\\\\\`call', lib/gitlab/middleware/request_context.rb:15:in \\\\\\\`call', lib/gitlab/middleware/webhook_recursion_detection.rb:15:in \\\\\\\`call', config/initializers/fix_local_cache_middleware.rb:11:in \\\\\\\`call', lib/gitlab/middleware/compressed_json.rb:44:in \\\\\\\`call', lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in \\\\\\\`call', lib/gitlab/middleware/sidekiq_web_static.rb:20:in \\\\\\\`call', lib/gitlab/metrics/requests_rack_middleware.rb:79:in \\\\\\\`call', lib/gitlab/middleware/release_env.rb:13:in \\\\\\\`call' \\\\\\\]

Strangely, Sentry points somewhat differently to:

What is the expected correct behavior?

Pipeline page loads without error.

Relevant logs and/or screenshots

See links in current bug behavior above ☝

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Results of GitLab application Check

Possible fixes

  1. Put a limit on number of components retrieved for the Licenses tab on the pipelines page (details).
  2. Update semver_dialects gem (details).

Proposed Implementation plans

  1. Limit number of components generated for the Licenses tab
  2. Update semver_dialects gem ( https://gitlab.com/gitlab-org/vulnerability-research/foss/semver_dialects/-/merge_requests/22, Bump semver_dialects gem version (!142607 - merged))
    1. Update String#number? to ensure that no exception is thrown when argument evaluated: lib/utils.rb
    2. Once new version of gem is published, open an MR in the rails repo to bump to this version: https://gitlab.com/gitlab-org/gitlab/blob/b60c867458f32461acbaf3298a2c4550382ed057/Gemfile#L276. MR: !142607 (merged)

Actual Implementation plan

  1. Disable haml template from calling license_scan_count(project, pipeline)
  2. Follow-up backend issue in backlog to improve backend performance #439925 (closed)
  3. For now, minor UI regression where the license tab won't show counts on page load, but clicking into the tab, will show the counts once the license tab content loads. UI issue captured here #440309
Edited by Fernando Cardenas