Auto-merge failed to merge pipeline
!121017 (merged) was set to auto-merge today, but it never merged.
In https://log.gprd.gitlab.net/goto/912b7c70-f98b-11ed-8afc-c9851e4645c0, I see AutoMergeProcessWorker
ran at 14:01, when the pipeline succeeded:
If I filter the correlation_id
(b0b48e2113e221b88abf3ce09cf7b941
) https://log.gprd.gitlab.net/goto/fb45e440-f98d-11ed-a017-0d32180b1390, I see this status:
"exception.backtrace": [
"lib/gitlab/git/wraps_gitaly_errors.rb:24:in `rescue in wrapped_gitaly_errors'",
"lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'",
"lib/gitlab/git/repository.rb:738:in `delete_refs'",
"app/models/repository.rb:906:in `delete_refs'",
"app/models/ci/persistent_ref.rb:14:in `delete_refs'",
"app/models/ci/persistent_ref.rb:30:in `delete'",
"app/models/ci/pipeline.rb:340:in `block (3 levels) in <class:Pipeline>'",
"app/models/concerns/after_commit_queue.rb:39:in `instance_eval'",
"app/models/concerns/after_commit_queue.rb:39:in `_run_after_commit_queue'",
"config/initializers/forbid_sidekiq_in_transactions.rb:106:in `block in committed!'",
"config/initializers/forbid_sidekiq_in_transactions.rb:10:in `skipping_transaction_check'",
"config/initializers/forbid_sidekiq_in_transactions.rb:106:in `committed!'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'",
"lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'",
"lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff'",
"lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'",
"lib/gitlab/database.rb:368:in `block in transaction'",
"lib/gitlab/database.rb:367:in `transaction'",
"app/models/concerns/cross_database_modification.rb:83:in `transaction'",
"lib/gitlab/optimistic_locking.rb:17:in `retry_lock'",
"app/models/ci/pipeline.rb:859:in `set_status'",
"app/services/ci/pipeline_processing/atomic_processing_service.rb:72:in `update_pipeline!'",
"app/services/ci/pipeline_processing/atomic_processing_service.rb:37:in `process!'",
"app/services/ci/pipeline_processing/atomic_processing_service.rb:23:in `block in execute'",
"app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'",
"app/services/ci/pipeline_processing/atomic_processing_service.rb:23:in `execute'",
"app/services/ci/process_pipeline_service.rb:16:in `execute'",
Somehow DeleteRefs
resulted in this Gitlab::Git::CommandError
:
13:unable to prepare: state update to \"prepare\" failed: EOF."
It appears the DeleteRefs
is a red herring; this exception is handled gracefully in https://gitlab.com/gitlab-org/gitlab/-/blob/72e0329c8d86e804f9ea152b590a73ce591a003e/app/services/auto_merge/merge_when_pipeline_succeeds_service.rb#L14-15.
@ayufan
mentioned that he's been seeing io: read/write on closed pipe
as well in https://gitlab.com/gitlab-org/gitlab/-/pipelines/875744705/failures. I wonder if there is some issue with Gitaly processes/nodes being shut down prematurely. UPDATE: This does not seem to be relevant here.