Merge requests get stuck in preparing state when NewMergeRequestWorker fails to complete

A few users in #263390 (comment 686786552) (and maybe @fzimmer) have noted that their merge requests are stuck with Merge request stuck on "checking if merge request can be merged...:

image

It appears that at least for two of these merge requests, they are in the preparing state:

[ gprd ] production> mr.mergeable?
=> false
[ gprd ] production> mr.mergeable_state?
=> true
[ gprd ] production> mr.recheck_merge_status?
=> false
[ gprd ] production> mr.merge_status
=> "preparing"
[ gprd ] production> mr.public_merge_status
=> "checking"

MergeRequest::AfterCreateService via NewMergeRequestWorker is supposed to kick this out of this state and into the unchecked state in https://gitlab.com/gitlab-org/gitlab/blob/b2e914991addb5275f8f5b8762634d33d4eca64e/app/services/merge_requests/after_create_service.rb#L7, but if this doesn't reach here we get into this state.

From https://log.gprd.gitlab.net/goto/cdea13891d44cde555445399330af3a1, we can see NewMergeRequestWorker failed due to a Gitaly timeout:

image

From https://log.gprd.gitlab.net/goto/68ff61fb3939fc4a6a4275db2eab149c, we see the GetBlobs gRPC call failed on file-39 after 10 seconds:

image

/cc: @patrickbajao

Edited by Stan Hu