Add Prometheus histogram for lock retries

What does this MR do?

Adds Prometheus histogram to track the number of retries happened during the executing of Optimistic Retry Lock.

It was suggested in #322125 (comment 520973774)

Does this MR meet the acceptance criteria?

Conformity

• 📋 Does this MR need a changelog?
  • I have included a changelog entry.
  • [-] I have not included a changelog entry because _____.
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• [-] Database guides
• [-] Separation of EE specific content

Availability and Testing

I tested locally like this.

1. I added an artificial exception in the retry lock code to trigger the retries

--- a/lib/gitlab/optimistic_locking.rb
+++ b/lib/gitlab/optimistic_locking.rb
@@ -11,6 +11,8 @@ def retry_lock(subject, max_retries = MAX_RETRIES, name:, &block)
       retry_attempts = 0

       begin
+        raise ActiveRecord::StaleObjectError if retry_attempts < 3
+
         ActiveRecord::Base.transaction do
           yield(subject)
         end

2. I opened one of the projects seeded in the dev database and restarted one of the pipelines

3. I checked the log tail -f log/service_measurement.log and verified there are records

{"severity":"INFO","time":"2021-03-03T12:06:34.546Z","correlation_id":"01EZVZQ2QSBCNQND108NSN5WG7","message":"Optimistic Lock released with retries","name":"ci_pipeline_set_status","retries":3,"time_s":0.060505000001285225}

4. I checked http://localhost:3000/-/metrics

# HELP gitlab_optimistic_locking_retries Multiprocess metric
# TYPE gitlab_optimistic_locking_retries histogram
gitlab_optimistic_locking_retries_bucket{le="+Inf"} 8
gitlab_optimistic_locking_retries_bucket{le="1"} 0
gitlab_optimistic_locking_retries_bucket{le="10"} 8
gitlab_optimistic_locking_retries_bucket{le="2"} 0
gitlab_optimistic_locking_retries_bucket{le="3"} 8
gitlab_optimistic_locking_retries_bucket{le="5"} 8
gitlab_optimistic_locking_retries_bucket{le="50"} 8
gitlab_optimistic_locking_retries_count 8
gitlab_optimistic_locking_retries_sum 24

Security

N/A

Related to #322125 (closed)