Record request retries.

What does this MR do?

Record request retries. A new counter metric has been introduced to record the number of retries.

Given that the retry client/requester is a few levels down, only readily available information useful for these metrics is the path and the method. Runner and System ID as not been passed for all methods and would require a bigger refactor. As such, for this iteration, we stick with the path and the method.

A normalized URL function is used to normalize the URL for the request, which helps reduce the cardinality of the prom metrics. The implementation of normalized URL is very simple, as the list of URLs to which requests are made only has integer values in the path. For a more complex URL, a REGEX-based approach would have been better, but it also comes with performance penalties (confirmed with benchmarks).

List of paths to which requests are made:

  1. Register runner: /runners
  2. Verify runner: /runners/verify
  3. Unregister runner: /runners
  4. Unregister runner manager: /runners/managers
  5. Reset token:
    1. /runners/reset_authentication_token
    2. /runners/{id}/reset_authentication_token
  6. Request jobs: /jobs/request
  7. Update jobs: /jobs/{id}
  8. Patch trace: /jobs/{id}/trace
  9. Upload artifacts: /jobs/{id}/artifacts
  10. Download artifacts: /jobs/{id}/artifacts

Metric output from the /metrics endpoint:

# HELP gitlab_runner_api_request_retries_total Total number of retries per endpoint
# TYPE gitlab_runner_api_request_retries_total counter
gitlab_runner_api_request_retries_total{method="patch",path="/api/v4/jobs/{id}/trace"} 0
gitlab_runner_api_request_retries_total{method="post",path="/api/v4/jobs/request"} 0
gitlab_runner_api_request_retries_total{method="put",path="/api/v4/jobs/{id}"} 0

Why was this MR needed?

To provide more visibility into request retries.

What's the best way to test this MR?

go test -race ./network/...

What are the relevant issue numbers?

Issue 38654

Edited by Love Bhardwaj

Merge request reports

Loading