Request timeouts during run_after_commit hooks can cause pipelines to get stuck in created

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When a pipeline creation request times out during run_after_commit hooks between Gitlab::Ci::Pipeline::Chain::Create and Gitlab::Ci::Pipeline::Chain::Pipeline::Process in the creation sequence, pipelines can get stuck in created.

See this issue on sentry.

Steps to reproduce

In GDK:

  1. Add very long sleep within Ci::Build#execute_hooks (long enough to trigger HTTP request timeouts)
  2. Try creating a pipeline using the API

Example Project

N/A, but I've seen this happening fairly frequently when creating merge request pipelines on gitlab-org/gitlab prior to setting an MR to auto-merge.

What is the current bug behavior?

The pipeline gets stuck in created

What is the expected correct behavior?

  1. Ideally, the request should simply not time out.
  2. Regardless, any exception before reaching the last step of the chain should probably fail the pipeline creation instead of bubbling up as an exception.

Relevant logs and/or screenshots

Rack::Timeout::RequestTimeoutException: Request ran for longer than 60000ms 
  from monitor.rb:202:in `synchronize'
  from monitor.rb:202:in `mon_synchronize'
  ...
  from app/models/ci/build.rb:465:in `retries_count'
  from lib/gitlab/data_builder/build.rb:22:in `build'
  from app/models/ci/build.rb:1147:in `block in build_data'
  from gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'
  from app/models/ci/build.rb:1147:in `build_data'
  from app/models/ci/build.rb:729:in `execute_hooks'
  from app/models/ci/build.rb:199:in `block (2 levels) in '
  from app/models/concerns/after_commit_queue.rb:44:in `instance_eval'
  from app/models/concerns/after_commit_queue.rb:44:in `_run_after_commit_queue'
  from activesupport (7.0.8.1) lib/active_support/callbacks.rb:400:in `block in make_lambda'
  from activesupport (7.0.8.1) lib/active_support/callbacks.rb:271:in `block in simple'
  from activesupport (7.0.8.1) lib/active_support/callbacks.rb:599:in `block in invoke_after'
  from activesupport (7.0.8.1) lib/active_support/callbacks.rb:599:in `each'
  from activesupport (7.0.8.1) lib/active_support/callbacks.rb:599:in `invoke_after'
  from activesupport (7.0.8.1) lib/active_support/callbacks.rb:108:in `run_callbacks'
  from activesupport (7.0.8.1) lib/active_support/callbacks.rb:929:in `_run_commit_callbacks'
  from activerecord (7.0.8.1) lib/active_record/transactions.rb:321:in `committed!'
  from config/initializers/skip_transaction_checks.rb:8:in `block (2 levels) in committed!'
  from config/initializers/forbid_sidekiq_in_transactions.rb:10:in `skipping_transaction_check'
  from config/initializers/skip_transaction_checks.rb:8:in `block in committed!'
  from lib/gitlab/exclusive_lease.rb:104:in `skipping_transaction_check'
  from config/initializers/skip_transaction_checks.rb:7:in `committed!'
  ...
  from lib/gitlab/database.rb:358:in `transaction'
  from app/models/concerns/cross_database_modification.rb:83:in `transaction'
  from activerecord (7.0.8.1) lib/active_record/transactions.rb:290:in `transaction'
  from lib/gitlab/ci/pipeline/chain/create.rb:15:in `block (3 levels) in perform!'
  from app/models/concerns/ci/bulk_insertable_tags.rb:13:in `with_bulk_insert_tags'
  from lib/gitlab/ci/pipeline/chain/create.rb:14:in `block (2 levels) in perform!'
  from app/models/concerns/bulk_insertable_associations.rb:54:in `with_bulk_insert'
  from lib/gitlab/ci/pipeline/chain/create.rb:13:in `block in perform!'
  from lib/gitlab/ci/pipeline/logger.rb:33:in `instrument'
  from lib/gitlab/ci/pipeline/logger.rb:43:in `instrument_once_with_sql'
  from lib/gitlab/ci/pipeline/chain/create.rb:12:in `perform!'
  from lib/gitlab/ci/pipeline/chain/sequence.rb:20:in `block in build!'
  from lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `each'
  from lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `build!'
  from app/services/ci/create_pipeline_service.rb:93:in `execute'
  from ee/app/services/ee/merge_requests/create_pipeline_service.rb:32:in `create_merged_result_pipeline_for'
  from ee/app/services/ee/merge_requests/create_pipeline_service.rb:10:in `execute'
  from lib/api/merge_requests.rb:579:in `block (2 levels) in '

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Results of GitLab application Check

Possible fixes

Edited by 🤖 GitLab Bot 🤖