### Summary A large ~"GitLab Ultimate" customer on ~SaaS experiences 500 errors for particular pipeline pages and generally "high load times on other\[s\]" in that project. Self-managed customers are affected as well, with reports from 16.5.1, 16.6.1 and 16.6.2. ### Steps to reproduce 1. Load pipeline page mentioned in [this internal support ticket](https://gitlab.zendesk.com/agent/tickets/479374) 2. Check our logs for the reported correlation ID, or for `json.exception.class: "Rack::Timeout::RequestTimeoutException" and json.path: "…their…project…"`. ### Example Project ### What is the current _bug_ behavior? 500 error page & [our Kibana logs](https://log.gprd.gitlab.net/app/r/s/Z8VAC) point to [version_in_default_licenses_range?@ee/app/models/package_metadata/package.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/v16.6.2-ee/ee/app/models/package_metadata/package.rb#L89-96). <details> <summary>Full json.exception.backtrace</summary> <pre> \\\\\\\[ ee/app/models/package_metadata/package.rb:96:in \\\\\\\`version_in_default_licenses_range?', ee/app/models/package_metadata/package.rb:84:in \\\\\\\`default_license_ids', ee/app/models/package_metadata/package.rb:34:in \\\\\\\`license_ids_for', ee/lib/gitlab/license_scanning/package_licenses.rb:48:in \\\\\\\`block (4 levels) in fetch', ee/lib/gitlab/license_scanning/package_licenses.rb:47:in \\\\\\\`each', ee/lib/gitlab/license_scanning/package_licenses.rb:47:in \\\\\\\`block (3 levels) in fetch', ee/lib/gitlab/license_scanning/package_licenses.rb:46:in \\\\\\\`block (2 levels) in fetch', ee/lib/gitlab/license_scanning/package_licenses.rb:43:in \\\\\\\`each_slice', ee/lib/gitlab/license_scanning/package_licenses.rb:43:in \\\\\\\`each', ee/lib/gitlab/license_scanning/package_licenses.rb:43:in \\\\\\\`block in fetch', lib/gitlab/database/load_balancing/session.rb:72:in \\\\\\\`use_replicas_for_read_queries', ee/lib/gitlab/license_scanning/package_licenses.rb:151:in \\\\\\\`use_replica_if_available', ee/lib/gitlab/license_scanning/package_licenses.rb:31:in \\\\\\\`fetch', ee/lib/gitlab/license_scanning/sbom_scanner.rb:16:in \\\\\\\`report', ee/app/helpers/ee/projects/pipeline_helper.rb:36:in \\\\\\\`license_scan_count', ee/app/helpers/ee/projects/pipeline_helper.rb:20:in \\\\\\\`js_pipeline_tabs_data', app/views/projects/pipelines/show.html.haml:31, app/controllers/application_controller.rb:134:in \\\\\\\`render', app/controllers/projects/pipelines_controller.rb:245:in \\\\\\\`block (2 levels) in render_show', app/controllers/projects/pipelines_controller.rb:243:in \\\\\\\`render_show', app/controllers/projects/pipelines_controller.rb:117:in \\\\\\\`block (2 levels) in show', app/controllers/projects/pipelines_controller.rb:116:in \\\\\\\`show', app/controllers/application_controller.rb:519:in \\\\\\\`block in allow_gitaly_ref_name_caching', lib/gitlab/gitaly_client.rb:472:in \\\\\\\`allow_ref_name_caching', app/controllers/application_controller.rb:518:in \\\\\\\`allow_gitaly_ref_name_caching', ee/lib/gitlab/ip_address_state.rb:10:in \\\\\\\`with', ee/app/controllers/ee/application_controller.rb:45:in \\\\\\\`set_current_ip_address', app/controllers/application_controller.rb:470:in \\\\\\\`set_current_admin', lib/gitlab/session.rb:11:in \\\\\\\`with_session', app/controllers/application_controller.rb:461:in \\\\\\\`set_session_storage', lib/gitlab/i18n.rb:114:in \\\\\\\`with_locale', lib/gitlab/i18n.rb:120:in \\\\\\\`with_user_locale', app/controllers/application_controller.rb:452:in \\\\\\\`set_locale', app/controllers/application_controller.rb:445:in \\\\\\\`set_current_context', ee/lib/omni_auth/strategies/group_saml.rb:41:in \\\\\\\`other_phase', lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in \\\\\\\`call', lib/gitlab/middleware/memory_report.rb:13:in \\\\\\\`call', lib/gitlab/middleware/speedscope.rb:13:in \\\\\\\`call', lib/gitlab/database/load_balancing/rack_middleware.rb:23:in \\\\\\\`call', lib/gitlab/middleware/rails_queue_duration.rb:33:in \\\\\\\`call', lib/gitlab/etag_caching/middleware.rb:21:in \\\\\\\`call', lib/gitlab/metrics/rack_middleware.rb:16:in \\\\\\\`block in call', lib/gitlab/metrics/web_transaction.rb:46:in \\\\\\\`run', lib/gitlab/metrics/rack_middleware.rb:16:in \\\\\\\`call', lib/gitlab/middleware/go.rb:20:in \\\\\\\`call', lib/gitlab/middleware/query_analyzer.rb:11:in \\\\\\\`block in call', lib/gitlab/database/query_analyzer.rb:37:in \\\\\\\`within', lib/gitlab/middleware/query_analyzer.rb:11:in \\\\\\\`call', lib/gitlab/middleware/multipart.rb:173:in \\\\\\\`call', lib/gitlab/middleware/read_only/controller.rb:50:in \\\\\\\`call', lib/gitlab/middleware/read_only.rb:18:in \\\\\\\`call', lib/gitlab/middleware/same_site_cookies.rb:27:in \\\\\\\`call', lib/gitlab/middleware/path_traversal_check.rb:35:in \\\\\\\`call', lib/gitlab/middleware/handle_malformed_strings.rb:21:in \\\\\\\`call', lib/gitlab/middleware/basic_health_check.rb:25:in \\\\\\\`call', lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in \\\\\\\`call', lib/gitlab/middleware/request_context.rb:15:in \\\\\\\`call', lib/gitlab/middleware/webhook_recursion_detection.rb:15:in \\\\\\\`call', config/initializers/fix_local_cache_middleware.rb:11:in \\\\\\\`call', lib/gitlab/middleware/compressed_json.rb:44:in \\\\\\\`call', lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in \\\\\\\`call', lib/gitlab/middleware/sidekiq_web_static.rb:20:in \\\\\\\`call', lib/gitlab/metrics/requests_rack_middleware.rb:79:in \\\\\\\`call', lib/gitlab/middleware/release_env.rb:13:in \\\\\\\`call' \\\\\\\] </pre> </details> Strangely, [Sentry](https://new-sentry.gitlab.net/organizations/gitlab/issues/173/events/474ff69a168547debf45b6a744f155ae/?project=3) points somewhat differently to: - [ee/lib/gitlab/license_scanning/package_licenses.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/v16.6.2-ee/ee/lib/gitlab/license_scanning/package_licenses.rb#L31) - [app/views/projects/pipelines/show.html.haml](https://gitlab.com/gitlab-org/gitlab/-/blob/v16.6.2-ee/app/views/projects/pipelines/show.html.haml#L31) ### What is the expected _correct_ behavior? Pipeline page loads without error. ### Relevant logs and/or screenshots See links in current _bug_ behavior above :point_up: ### Output of checks This bug happens on GitLab.com #### ~~Results of GitLab environment info~~ #### ~~Results of GitLab application Check~~ ### Possible fixes 1. Put a limit on number of components retrieved for the `Licenses` tab on the pipelines page ([details](https://gitlab.com/gitlab-org/gitlab/-/issues/435272#note_1735216681 "Rack::Timeout::RequestTimeoutException on particular pipeline pages")). 2. Update semver_dialects gem ([details](https://gitlab.com/gitlab-org/gitlab/-/issues/435272#note_1735216350 "Rack::Timeout::RequestTimeoutException on particular pipeline pages")). ### Proposed Implementation plans 1. Limit number of components generated for the `Licenses` tab * Update the `PackageLicenses#fetch` to break the number of components exceeds specified amount (if set): https://gitlab.com/gitlab-org/gitlab/-/blob/ba3da1fcec222dd6a5eae9513a30036df047ac33/ee/lib/gitlab/license_scanning/package_licenses.rb#L43 * Update `SbomScanner` to accept this value and pass it to `PackageLicenses`: https://gitlab.com/gitlab-org/gitlab/blob/81017f878fc98f4207e0e4bb0dafd0ea5027a18f/ee/lib/gitlab/license_scanning/sbom_scanner.rb#L5 * Update the pipeline helper to pass this value to `SbomScanner`: https://gitlab.com/gitlab-org/gitlab/blob/f6f64deab7c120a197ab0260dbb77b74182b0e57/ee/app/helpers/ee/projects/pipeline_helper.rb#L36 2. Update `semver_dialects` gem ( https://gitlab.com/gitlab-org/vulnerability-research/foss/semver_dialects/-/merge_requests/22, https://gitlab.com/gitlab-org/gitlab/-/merge_requests/142607+) 1. Update `String#number?` to ensure that no exception is thrown when argument evaluated: [lib/utils.rb](https://gitlab.com/gitlab-org/vulnerability-research/foss/semver_dialects/-/blob/3e456b533ecd88625d9f7a85022c42e2db1b3624/lib/utils.rb#L18) 2. Once new version of gem is published, open an MR in the rails repo to bump to this version: https://gitlab.com/gitlab-org/gitlab/blob/b60c867458f32461acbaf3298a2c4550382ed057/Gemfile#L276. MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/142607 ### Actual Implementation plan 1. Disable haml template from calling `license_scan_count(project, pipeline)` 1. Follow-up backend issue in backlog to improve backend performance https://gitlab.com/gitlab-org/gitlab/-/issues/439925 1. For now, minor UI regression where the license tab won't show counts on page load, but clicking into the tab, will show the counts once the license tab content loads. UI issue captured here https://gitlab.com/gitlab-org/gitlab/-/issues/440309