Request timeouts during run_after_commit hooks can cause pipelines to get stuck in created
Summary
When a pipeline creation request times out during run_after_commit
hooks between Gitlab::Ci::Pipeline::Chain::Create
and Gitlab::Ci::Pipeline::Chain::Pipeline::Process
in the creation sequence, pipelines can get stuck in created
.
See this issue on sentry.
Steps to reproduce
In GDK:
- Add very long
sleep
withinCi::Build#execute_hooks
(long enough to trigger HTTP request timeouts) - Try creating a pipeline using the API
Example Project
N/A, but I've seen this happening fairly frequently when creating merge request pipelines on gitlab-org/gitlab
prior to setting an MR to auto-merge.
What is the current bug behavior?
The pipeline gets stuck in created
What is the expected correct behavior?
- Ideally, the request should simply not time out.
- Regardless, any exception before reaching the last step of the chain should probably fail the pipeline creation instead of bubbling up as an exception.
Relevant logs and/or screenshots
Rack::Timeout::RequestTimeoutException: Request ran for longer than 60000ms from monitor.rb:202:in `synchronize' from monitor.rb:202:in `mon_synchronize' ... from app/models/ci/build.rb:465:in `retries_count' from lib/gitlab/data_builder/build.rb:22:in `build' from app/models/ci/build.rb:1147:in `block in build_data' from gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize' from app/models/ci/build.rb:1147:in `build_data' from app/models/ci/build.rb:729:in `execute_hooks' from app/models/ci/build.rb:199:in `block (2 levels) in ' from app/models/concerns/after_commit_queue.rb:44:in `instance_eval' from app/models/concerns/after_commit_queue.rb:44:in `_run_after_commit_queue' from activesupport (7.0.8.1) lib/active_support/callbacks.rb:400:in `block in make_lambda' from activesupport (7.0.8.1) lib/active_support/callbacks.rb:271:in `block in simple' from activesupport (7.0.8.1) lib/active_support/callbacks.rb:599:in `block in invoke_after' from activesupport (7.0.8.1) lib/active_support/callbacks.rb:599:in `each' from activesupport (7.0.8.1) lib/active_support/callbacks.rb:599:in `invoke_after' from activesupport (7.0.8.1) lib/active_support/callbacks.rb:108:in `run_callbacks' from activesupport (7.0.8.1) lib/active_support/callbacks.rb:929:in `_run_commit_callbacks' from activerecord (7.0.8.1) lib/active_record/transactions.rb:321:in `committed!' from config/initializers/skip_transaction_checks.rb:8:in `block (2 levels) in committed!' from config/initializers/forbid_sidekiq_in_transactions.rb:10:in `skipping_transaction_check' from config/initializers/skip_transaction_checks.rb:8:in `block in committed!' from lib/gitlab/exclusive_lease.rb:104:in `skipping_transaction_check' from config/initializers/skip_transaction_checks.rb:7:in `committed!' ... from lib/gitlab/database.rb:358:in `transaction' from app/models/concerns/cross_database_modification.rb:83:in `transaction' from activerecord (7.0.8.1) lib/active_record/transactions.rb:290:in `transaction' from lib/gitlab/ci/pipeline/chain/create.rb:15:in `block (3 levels) in perform!' from app/models/concerns/ci/bulk_insertable_tags.rb:13:in `with_bulk_insert_tags' from lib/gitlab/ci/pipeline/chain/create.rb:14:in `block (2 levels) in perform!' from app/models/concerns/bulk_insertable_associations.rb:54:in `with_bulk_insert' from lib/gitlab/ci/pipeline/chain/create.rb:13:in `block in perform!' from lib/gitlab/ci/pipeline/logger.rb:33:in `instrument' from lib/gitlab/ci/pipeline/logger.rb:43:in `instrument_once_with_sql' from lib/gitlab/ci/pipeline/chain/create.rb:12:in `perform!' from lib/gitlab/ci/pipeline/chain/sequence.rb:20:in `block in build!' from lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `each' from lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `build!' from app/services/ci/create_pipeline_service.rb:93:in `execute' from ee/app/services/ee/merge_requests/create_pipeline_service.rb:32:in `create_merged_result_pipeline_for' from ee/app/services/ee/merge_requests/create_pipeline_service.rb:10:in `execute' from lib/api/merge_requests.rb:579:in `block (2 levels) in '
Output of checks
This bug happens on GitLab.com
Results of GitLab environment info
Results of GitLab application Check
Possible fixes
Edited by Hordur Freyr Yngvason