OverrideUuidsService is raising `PG::QueryCanceled`
Why are we doing this work
OverrideUuidsService
is raising PG::QueryCanceled
which has to be fixed.
Relevant links
- Related sentry issue: https://sentry.gitlab.net/gitlab/gitlabcom/issues/4090978
Exception backtrace
Backtrace
"lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `public_send'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `block in read_using_load_balancer'",
"lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'",
"lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff'",
"lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `read_using_load_balancer'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'",
"ee/app/services/security/override_uuids_service.rb:95:in `existing_signatures'",
"ee/app/services/security/override_uuids_service.rb:71:in `existing_finding_by_signature'",
"ee/app/services/security/override_uuids_service.rb:57:in `override_uuid_for'",
"ee/app/services/security/override_uuids_service.rb:49:in `block in execute'",
"ee/app/services/security/override_uuids_service.rb:49:in `each'",
"ee/app/services/security/override_uuids_service.rb:49:in `execute'",
"ee/app/services/security/override_uuids_service.rb:38:in `execute'",
"ee/app/services/security/override_uuids_service.rb:25:in `block in execute'",
"ee/app/services/security/override_uuids_service.rb:25:in `each'",
"ee/app/services/security/override_uuids_service.rb:25:in `each_slice'",
"ee/app/services/security/override_uuids_service.rb:25:in `execute'",
"ee/app/services/security/override_uuids_service.rb:14:in `execute'",
"ee/app/services/security/store_scan_service.rb:43:in `override_finding_uuids!'",
"ee/app/services/security/store_scan_service.rb:24:in `execute'",
"ee/app/services/security/store_scan_service.rb:14:in `execute'",
"ee/app/services/security/store_grouped_scans_service.rb:60:in `store_scan_for'",
"ee/app/services/security/store_grouped_scans_service.rb:23:in `block (2 levels) in execute'",
"ee/app/services/security/store_grouped_scans_service.rb:22:in `each'",
"ee/app/services/security/store_grouped_scans_service.rb:22:in `reduce'",
"ee/app/services/security/store_grouped_scans_service.rb:22:in `block in execute'",
"lib/gitlab/exclusive_lease_helpers.rb:38:in `in_lock'",
"ee/app/services/security/store_grouped_scans_service.rb:21:in `execute'",
"ee/app/services/security/store_grouped_scans_service.rb:12:in `execute'",
"ee/app/services/security/store_scans_service.rb:16:in `block in execute'",
"ee/app/services/security/store_scans_service.rb:16:in `each'",
"ee/app/services/security/store_scans_service.rb:16:in `execute'",
"ee/app/services/security/store_scans_service.rb:6:in `execute'",
"ee/app/workers/security/store_scans_worker.rb:21:in `block in perform'",
"ee/app/workers/security/store_scans_worker.rb:16:in `perform'",
"lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'",
"lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'",
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call'",
"lib/gitlab/application_context.rb:115:in `block in use'",
"lib/gitlab/application_context.rb:115:in `use'",
"lib/gitlab/application_context.rb:55:in `with_context'",
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call'",
"lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'",
"lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:37:in `within'",
"lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'",
"lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'",
"lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
"lib/gitlab/with_request_store.rb:10:in `with_request_store'",
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'",
"lib/gitlab/metrics/background_transaction.rb:33:in `run'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'",
"lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'",
"lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'",
"lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'",
"lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'"
Implementation plan
As per @minac's comment below:
Seems like there is no index on
signature_sha
which is used by the timing-out query. Adding an index for that column can fix this issue.
Adding an index for vulnerability_finding_signatures
on signature_sha
can fix the timeout.
While the table size is big, the index can be created concurrently in a post-deployment migration.
Please check this comment for more details.
Verification steps
-
Check if the error does not happen anymore.
Edited by Ahmed Hemdan