Creates a gitlab_runner_cache_s3_assume_role_failures_total metric

What does this MR do?

This MR creates a metric gitlab_runner_cache_s3_assume_role_failures_total which counts the number of failed attempts to assume a role.

PS: Usually I would have called a new metric a type::feature but I marked it as type::maintainence instead because I was following what was done in this merge request !6345 (merged). If thats wrong, let me know :)

Why was this MR needed?

I'm looking for a metric I can use to alert on if the IMDS throttling on AssumeRole requests returns. Of course, I could have just alerted on the presence of failed to assume role or failed to retrieve credentials in the logs - but a Prometheus metric will integrate much better with the Dedicated Hosted Runners QA testing.

I'm going to say a spike in gitlab_runner_cache_s3_assume_role_failures_total as a percentage of gitlab_runner_cache_s3_assume_role_cache_misses_total will be a decent indicator of the possibility of IMDS throttling.

What's the best way to test this MR?

I added a new test TestFetchCredentialsForRole_FailureMetric which runs in the MR CICD pipeline as part of the integration tests. It tests both pathways - Failed to assume role for cache credentials and AssumeRole succeeded but returned no credentials

My new test more or less follows the existing patterns of other *_Metrics tests in cache/s3v2/s3_test.go

What are the relevant issue numbers?

https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/work_items/11910

Also Stan and I chatted about this idea here - #39327 (comment 3236544337)

Edited by Zoe Braddock

Merge request reports

Loading