[Spike] Backend: Transient: MR added to merge train doesn't get merged and fails silently

Overview / Problem

As a follow-up to gitlab-com/www-gitlab-com!81915 (merged), the activity shows that the MR was added to the merge train on 2021-May-12, however remained unmerged.

On 2021-May-20, it was removed and added back to the Merge Train and it merged successfully.

The purpose of this issue is to investigate the bug and try to reproduce the behaviour from the referenced MR, then propose a solution to resolve this transient behaviour. If the findings suggest that we need to make adjustments to the merge train experience, new issue would be created for the same.

Proposal

1. This Issue: Gather data through kibana logs to look for this issue / find possible causes
2. This Issue: Create weighted issue/issues with a technical proposal to fix the bug

---

1. Out of scope/Future issue: Implement fix based on the top identified issues from kibana logs that were reproducible and/or top 50% of observed cases