UpdateMergeRequestsWorker lock contention

UpdateMergeRequestsWorker is experiencing a lot of lock contention

Slowlog search: https://log.gprd.gitlab.net/goto/847ddc7a734c14c85f5ac2c915cd385a

This in turn is saturation pgbouncer: https://dashboards.gitlab.net/d/pgbouncer-main/pgbouncer-overview?orgId=1&from=1618306200000&to=1618327859999&var-PROMETHEUS_DS=Global&var-environment=gprd&var-sigma=2&viewPanel=1228921753

Which is leading to a rails_sql SLO violation: https://gitlab.slack.com/archives/C101F3796/p1618327652181400

SQL Statement

Contention is occurring for

UPDATE "merge_requests" SET "merge_status" = $1 WHERE "merge_requests"."id" = $2

Client Transactions

One of the problems appears to be that UpdateMergeRequestsWorker relies on client transactions that are being held open and leading to the contention.

(Placeholder - more details to follow)