Instrument our optimistic locking framework with Logging and Prometheus metrics
Description
Gitlab::OptimisticLocking#retry_lock
should export counters to record the number of attempts at a retry, and total time spend attempting update.
In a change from 2019, 56ae34e4, I noted that Optimistic Locking in Gitlab::OptimisaticLocking#retry_lock
does not report on the number of retry attempts it performs.
https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/optimistic_locking.rb#L9
The code allows us to 100 attempts. If, for some reason, we are attempting many retries, this will put a great deal of pressure on our primary database, but will not be immediately apparent, or reported.
This code is primarily used as part of CI.
Proposal
- We should add a counter to report on the total number of retries.
- We should record the total duration spent attempting to perform the update.
- Make each
retry_lock
to be named to understand where it is coming from
cc @grzesiek