Merge request on merge train fails to merge

Summary

In https://gitlab.com/gitlab-com/dev-sub-department/section-dev-request-for-help/-/issues/98 (internal), we got a report that a customer is sometimes experiencing a MR in a merge train failing to merge. The MR is mergeable though.

Steps to reproduce

Based on the report, it seems to be just adding a MR to merge train and wait for it to be merged. It doesn't happen all the time though.

What is the current bug behavior?

MR that is mergeable fails to merge and gets removed from merge train.

What is the expected correct behavior?

MR gets merged since it's mergeable.

Possible fixes

Based on investigation done in https://gitlab.com/gitlab-com/dev-sub-department/section-dev-request-for-help/-/issues/98#note_1446694103:

1. Add more logging when MergeRequests::MergeabilityCheckService is executed and when lock is acquired or not acquired. This should help us debug issues like this further.
2. When calling MergeRequest#mergeable? in MergeService via merge trains, we allow it to retry checking for lock. This may help if the case I mentioned above is true since this will ensure that we're able to check for mergeability before we call MergeRequest#can_be_merged?.