Backend: ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout
https://sentry.gitlab.net/gitlab/gitlabcom/issues/959944/
PG::QueryCanceled: ERROR: canceling statement due to statement timeout
active_record/connection_adapters/postgresql_adapter.rb:611:in `async_exec_params'
@connection.exec_params(sql, type_casted_binds)
active_record/connection_adapters/postgresql_adapter.rb:611:in `block (2 levels) in exec_no_cache'
@connection.exec_params(sql, type_casted_binds)
active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
yield
active_support/concurrency/share_lock.rb:187:in `yield_shares'
yield
active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
@lock.yield_shares(compatible: [:load]) do
...
(155 additional frame(s) were not displayed)
ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout
: SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" IN (SELECT max(id) FROM "ci_pipelines" WHERE "ci_pipelines"."project_id" = 13172553 AND "ci_pipelines"."ref" IN ('1028-refactor-gigsmart-corporate-notifications', '1065-refreshing-web-should-not-redirect-home', '1074-add-email-phone-validation', '1109/close-gig-event-tracking', '1111/toast-background-update', '1122-beta-camera-fixes', '1138-not-provided', '1157/flow-updates', '1157/fomu-addition', '1157/katana-updates', '1157/step-1', '1180/resize-worker-logout-button', '1184/message-time-sent-text', '1187/bugfix/worker-row-skill-match', '1187/skills-fixes', '1210/date-time-picker-ios-13', '1210/datetime-picker-ios-13', '1221-get-gigs-worker-that-is-tapped-to-be-hired-on-verified-gig-gets-gig-state-changed-to-confirming-via-hire-notification', '1222/bugfix/skill-select-freezing', '1223/populate-gig-status', '1228/gig-wizard-onboarding-modal', '126-requester-onboarding-wizard-web', '127-right-app-modal', '182/toggle-application-status-gig-action', '357-application-questionaire', '364-credit-card-web', '367-ie11-issues', '992/bugfix/gig-wizard-routing', 'ADMIN-46', 'ADMIN-55', 'ALC-614-no-tests', 'ALC-715-textinput-format', 'ALC-775/requester-notification-routing-tmp', 'ALC-791/prebuild-api-image', 'ALC-792/fix-update-overrides', 'ALC-872/iosBadge', 'ALC-920-gig-success-screen', 'ALC-943-notification-icon-bold-worker', 'BMS-1/bugfix/end-time-vs-date', 'BMS-123/zero-state-tabs', 'BMS-134/gig-receipt-email-variables-2', 'BMS-134/gig-receipt-emails', 'BMS-134/gig-receipt-fee-math', 'BMS-140-notification-mark-as-read', 'BMS-146/pause-requested-gwstate', 'BMS-146/pause-requested-gwstate-2', 'BMS-147/inactive-gig-worker-roster', 'BMS-175/worker-profile-row', 'BMS-176/update-skill-category-name', 'BMS-184-requester-org-nudge', 'BMS-184-requester-org-nudge--TEST', 'BMS-222/create-gig-payment-screen', 'BMS-227/create-gig-address-screen', 'BMS-231/create-gig-details-screen', 'BMS-236/worker-count-display', 'BMS-239/edit-from-summary', 'BMS-246/read-all-messages', 'BMS-252-update-messages-badge', 'BMS-253/ensure-worker-has-time-zone', 'BMS-255-darren', 'BMS-258-worker-row-status-colors', 'BMS-262/fix-worker-routes-bug', 'BMS-265/bugfix/worker-skill-hourly-rate', 'BMS-265/bugfix/worker-skill-hourly-rate-beta', 'BMS-265/bugfix/worker-skill-hourly-rate-dev', 'BMS-265/bugfix/worker-skill-hourly-rate-validation', 'BMS-281-ios-deeplink-fix', 'BMS-289-new-onboarding', 'BMS-289-requester-onboarding', 'BMS-293', 'BMS-294/notifications-permissions', 'BMS-299-max-line-count-messaging', 'BMS-306-remove-worker', 'BMS-306/decline-applicant-beta', 'BMS-319/bugfix/skill-search-limit', 'BMS-319/bugfix/skill-search-limit-prod', 'BMS-333/update-onboarding-locations', 'BMS-335-ios-industry-predictive-input', 'BMS-345/bugfix/archive-asap-inactive-workers-at-endtime', 'BMS-360-get-gigs-new-onboarding-fix', 'BMS-364/notification-query-fixes', 'BMS-366-route-from-history', 'BMS-367-navigation-loop', 'BMS-368-remove-routing', 'BMS-369/update-gig-type-button', 'BMS-370-new-onboarding-adjustments', 'BMS-370-new-onboarding-adjusts', 'BMS-374/requester-faq-link', 'BMS-376-hire-button-logic', 'BMS-396-application-questionnaire-worker', 'BMS-396/application-questionnaire', 'BMS-396/routing-fixes', 'BMS-397-application-questionnaire-requester', 'BMS-400/archive-messages-24-hours', 'BMS-400/cleanup-gig-service-worker', 'BMS-409/add-time-zone-to-gig', 'BMS-427/search-and-track-gigs', 'BMS-428/search-results-query-multi-skills', 'BMS-430/admin-schema-messages', 'BMS-432/email-environment-variables', 'BMS-445/bugfix/worker-gig-receipts', 'BMS-451/clear-out-old-gig-data', 'BMS-451/reset-gig', 'BMS-457/bugfix/notifications-after-canceled', 'BMS-463_auto_better_spelling', 'BMS-482/replace-gig-details-screen', 'BMS-508/bugfix/receipt-template-reference', 'BMS-536/pause-request-view-profile-row', 'MACH-556/bugfix-e2e', 'MACH-568/bugfix/sendgrid-email-details-fixes', 'MACH-644-search-skills-scroll-fix', 'MACH-742/bugfix-web-resiz...
ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout
: SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" IN (SELECT max(id) FROM "ci_pipelines" WHERE "ci_pipelines"."project_id" = 13172553 AND "ci_pipelines"."ref" IN ('1028-refactor-gigsmart-corporate-notifications', '1065-refreshing-web-should-not-redirect-home', '1074-add-email-phone-validation', '1109/close-gig-event-tracking', '1111/toast-background-update', '1122-beta-camera-fixes', '1138-not-provided', '1157/flow-updates', '1157/fomu-addition', '1157/katana-updates', '1157/step-1', '1180/resize-worker-logout-button', '1184/message-time-sent-text', '1187/bugfix/worker-row-skill-match', '1187/skills-fixes', '1210/date-time-picker-ios-13', '1210/datetime-picker-ios-13', '1221-get-gigs-worker-that-is-tapped-to-be-hired-on-verified-gig-gets-gig-state-changed-to-confirming-via-hire-notification', '1222/bugfix/skill-select-freezing', '1223/populate-gig-status', '1228/gig-wizard-onboarding-modal', '126-requester-onboarding-wizard-web', '127-right-app-modal', '182/toggle-application-status-gig-action', '357-application-questionaire', '364-credit-card-web', '367-ie11-issues', '992/bugfix/gig-wizard-routing', 'ADMIN-46', 'ADMIN-55', 'ALC-614-no-tests', 'ALC-715-textinput-format', 'ALC-775/requester-notification-routing-tmp', 'ALC-791/prebuild-api-image', 'ALC-792/fix-update-overrides', 'ALC-872/iosBadge', 'ALC-920-gig-success-screen', 'ALC-943-notification-icon-bold-worker', 'BMS-1/bugfix/end-time-vs-date', 'BMS-123/zero-state-tabs', 'BMS-134/gig-receipt-email-variables-2', 'BMS-134/gig-receipt-emails', 'BMS-134/gig-receipt-fee-math', 'BMS-140-notification-mark-as-read', 'BMS-146/pause-requested-gwstate', 'BMS-146/pause-requested-gwstate-2', 'BMS-147/inactive-gig-worker-roster', 'BMS-175/worker-profile-row', 'BMS-176/update-skill-category-name', 'BMS-184-requester-org-nudge', 'BMS-184-requester-org-nudge--TEST', 'BMS-222/create-gig-payment-screen', 'BMS-227/create-gig-address-screen', 'BMS-231/create-gig-details-screen', 'BMS-236/worker-count-display', 'BMS-239/edit-from-summary', 'BMS-246/read-all-messages', 'BMS-252-update-messages-badge', 'BMS-253/ensure-worker-has-time-zone', 'BMS-255-darren', 'BMS-258-worker-row-status-colors', 'BMS-262/fix-worker-routes-bug', 'BMS-265/bugfix/worker-skill-hourly-rate', 'BMS-265/bugfix/worker-skill-hourly-rate-beta', 'BMS-265/bugfix/worker-skill-hourly-rate-dev', 'BMS-265/bugfix/worker-skill-hourly-rate-validation', 'BMS-281-ios-deeplink-fix', 'BMS-289-new-onboarding', 'BMS-289-requester-onboarding', 'BMS-293', 'BMS-294/notifications-permissions', 'BMS-299-max-line-count-messaging', 'BMS-306-remove-worker', 'BMS-306/decline-applicant-beta', 'BMS-319/bugfix/skill-search-limit', 'BMS-319/bugfix/skill-search-limit-prod', 'BMS-333/update-onboarding-locations', 'BMS-335-ios-industry-predictive-input', 'BMS-345/bugfix/archive-asap-inactive-workers-at-endtime', 'BMS-360-get-gigs-new-onboarding-fix', 'BMS-364/notification-query-fixes', 'BMS-366-route-from-history', 'BMS-367-navigation-loop', 'BMS-368-remove-routing', 'BMS-369/update-gig-type-button', 'BMS-370-new-onboarding-adjustments', 'BMS-370-new-onboarding-adjusts', 'BMS-374/requester-faq-link', 'BMS-376-hire-button-logic', 'BMS-396-application-questionnaire-worker', 'BMS-396/application-questionnaire', 'BMS-396/routing-fixes', 'BMS-397-application-questionnaire-requester', 'BMS-400/archive-messages-24-hours', 'BMS-400/cleanup-gig-service-worker', 'BMS-409/add-time-zone-to-gig', 'BMS-427/search-and-track-gigs', 'BMS-428/search-results-query-multi-skills', 'BMS-430/admin-schema-messages', 'BMS-432/email-environment-variables', 'BMS-445/bugfix/worker-gig-receipts', 'BMS-451/clear-out-old-gig-data', 'BMS-451/reset-gig', 'BMS-457/bugfix/notifications-after-canceled', 'BMS-463_auto_better_spelling', 'BMS-482/replace-gig-details-screen', 'BMS-508/bugfix/receipt-template-reference', 'BMS-536/pause-request-view-profile-row', 'MACH-556/bugfix-e2e', 'MACH-568/bugfix/sendgrid-email-details-fixes', 'MACH-644-search-skills-scroll-fix', 'MACH-742/bugfix-web-resize', 'MACH-769/hamburger-menu-mobile-beta', 'MACH-802/bugfix-worker-sms-confirmation', 'MACH-890-fix-firebase-ios', 'MACH-978/bugfix/time-diff-validation', 'MACH-981/available-message-actions', 'MACH-988/message-subscription', 'MBS/437_gigs_created_text_change', 'VADR-1001-fix-sad-panda-on-gig-creation', 'VADR-107-sms-sconfirmation-web-design', 'VADR-109-requester-profile-update', 'VADR-1095-fix-y-scroll', 'VADR-1098-fix-home-page-overflow-on-buttons', 'VADR-1108-IE-modal', 'VADR-111-create-gig-info-cards', 'VADR-117-support-page-web', 'VADR-119-action-button-redesign', 'VADR-120-secondary-action-button', 'VADR-121-add-back-buttons', 'VADR-122-worker-row-underlined', 'VADR-124-requester-view-worker-badge-photo', 'VADR-126-credit-card-input-screen', 'VADR-139-Active-worker-states', 'VADR-140-history-page-design', 'VADR-143-back-button-in-browser-mobile', 'VADR-147-background-card', 'VADR-149-design-update-profile-onboarding', 'VADR-151-Fix-rings-animation', 'VADR-155-redesign-profile-account-screen', 'VADR-159-ratings-and-reviews', 'VADR-160-introduce-hover-over', 'VADR-167-hover-over-card-element', 'VADR-169-missing-payment-modal', 'VADR-174-create-gig-hourly-rate', 'VADR-175-gig-wizard-address', 'VADR-176-gig-creation-progress-bar', 'VADR-178-logo-to-home', 'VADR-179-add-side-nav-to-all-onboarding-pages', 'VADR-180-share-screen', 'VADR-182-design-update-start-end-time', 'VADR-184-gig-details-text-page', 'VADR-188-processing-fee-gig-description-modals', 'VADR-201-new-styled-picker', 'VADR-203-side-menu-update', 'VADR-205-gig-summary', 'VADR-208-feature-flags', 'VADR-217-requester-view-worker-profile', 'VADR-221/worker-employment-web', 'VADR-233-time-extension-screen', 'VADR-235-active-worker-design-update', 'VADR-238-referral-announcement-notifiction', 'VADR-240-history-gigworker', 'VADR-249', 'VADR-26-skill-category-design', 'VADR-267-gig-card-spacing', 'VADR-271-place-badge-count-on-icon', 'VADR-271-place-badge-count-on-message-icon', 'VADR-278-show-build-info', 'VADR-279-sad-panda-reload-on-web', 'VADR-28-sms-confirmation', 'VADR-3-browse-workers-page', 'VADR-3-onboarding-browse-workers-profile-card', 'VADR-327-design-update-date-selection', 'VADR-375-design-only-worker-rows-layout', 'VADR-4-onboarding-sign-up', 'VADR-49-e2e-testing', 'VADR-62-katana-nav-stories', 'VADR-69-storybook-katana-web-alert', 'VADR-70-storybook-katana-container', 'VADR-91-Refactor-confirm-propmt', 'VADR-94-web-side-nav', 'about-me-test-fix', 'add_errors_to_photo_upload', 'adding-some-hooks', 'admin/add-blocking', 'admin/application-questionnaires', 'admin/fix-row-styles', 'alc-925-gig-does-not-auto-end', 'alpha', 'api-capabilities', 'base-topic-changes', 'beta', 'beta-127-right-app-modal', 'beta-backmerge-to-dev', 'bms-238-datepicker', 'bms-311-worker-count', 'bms-346-messaging-connectivity', 'bms-354-getworkers', 'bms-494-gig-state-query', 'browserstack-espresso', 'browserstack-touch-scroll', 'browserstack-working', 'bug/fix-about-page', 'bug/fix-early-preauth', 'bug/gig-not-ending-with-extension', 'bug/hired-workers-default-text-on-canceled-gig', 'bug/mobile-number-resolution', 'bug/twilio-fixes', 'bugfix/1002/multi-skill-modals', 'bugfix/lint-error', 'bugfix/mutations-in-cypress', 'bugfix/referral-log-out-dev', 'cherry-pick-44f6aed4', 'cherry-pick-7d8c5543', 'ci-screenshot-helper', 'ci/change_docker_build_base_dir', 'ci/e2e-tests-on-mac', 'ci/fix_android_docker_build', 'ci/improve-caching', 'ci/remove_npm_change', 'ci/test_mac_mini', 'ci/update_gradle', 'ci/use_dir_pattern_for_android', 'container-header-refactor', 'coveralls', 'data/app_assoc_event', 'dev', 'dev-analytics-update', 'devex/88_remove_events', 'devex/dont_upgrade_pods', 'devex/fix_requester', 'devex/huskey_test', 'devex/jest_retry', 'devex/mac_improvements', 'devex/reenable_dag', 'devex/remove_crashlytics', 'devex/remove_jest_circus', 'devex/what_i_pushed_to_prod_from_local', 'e2e-vadr81-update-skills', 'e2e/MACH-819/multiple-skills-pom-refactor', 'e2e/MACH-821/worker-onboarding', 'e2e/MACH-821/worker-onboarding-2', 'e2e/change-dirs', 'e2e/e2e-e2e-e2e
Examples (GitLab Internal)
Implementation Guide
From #34819 (comment 234933115)
This bug is localized here: https://gitlab.com/gitlab-org/gitlab/blob/c24432ee6f1103b0cf7ceab6a4da07f954ceec68/app/finders/pipelines_finder.rb#L64
When we scope: 'tags'
we filter all pipelines by ref: project.repository.tag_names
which pull all available tag names.
We are hitting this timeouts with tags but I think we could hit the same with branches as we do the same.
Perhaps we could have ci_pipelines.ref_type
to be an enum: [:tag, :branch, :merge_request, :unknown]
and when scope: 'tags'
we filter by ref_type
. This would also reduce Gitaly calls as we pull those refs from Gitaly.
We need to consider how we migrate existing data, perhaps we can keep the legacy check against Gitaly as fallback if ref_type
is nil
(or default :unknown
) and then update the pipeline.ref_type
with the result from Gitaly.