Expose circuit breaker values in workhorse config

What does this MR do?

Expose circuit breaker values in workhorse config

This allows configuration of the workhorse circuit breaker per
release environment. This supports deploying the circuit breaker
on gitlab.com

Changelog: added

The circuit breaker is implemented in workhorse, and has a dependency on redis.

The intention here is to apply the circuit breaker only to requests made by gitlab-shell to rails' internal API. These requests are proxied by workhorse. Certain aspects of the circuit breaker are not able to be locked behind a feature flag, so I intend to enable the circuit breaker in the config on staging, and monitoring prior to enabling it on production.

Additional documentation for the circuit breaker, and its implementation issues can be found in the epic https://gitlab.com/groups/gitlab-org/-/epics/19047+

The corresponding k8s-workloads/gitlab.com MR: Enable workhorse circuitbreaker in gstg (gitlab-com/gl-infra/k8s-workloads/gitlab-com!4684 - merged)

I'm not sure how to evaluate the following checklist item: test plan for scenarios not covered by automated tests. Please advise if there's any additional verification I can do.

Test Plan

After this is deployed to staging:

• Logs indicate workhorse successfully connected to redis
• No logs indicating the circuit breaker failed to be created
• Make repeated git operations via ssh until rate limited (600 requests per minute)
• Continue making requests
• Requests fail, and circuit breaker logs show requests have been circuit broken (state change, request denied with user key, etc.)
• Wait the duration of the timeout
• Requests succeed
• Make repeated git operations via ssh until rate limited (600 requests per minute)
• Wait the duration of the interval
• Circuit breaker state in redis resets to default
• Wait the duration of the redis key expiry (30 minutes)
• Circuit breaker state in redis is no longer present

Related issues

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/541105

Author checklist

For general guidance, please follow our Contributing guide.

Required

For anything in this list which will not be completed, please provide a reason in the MR discussion.

• Merge Request Title and Description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• MR has a green pipeline.
• Documentation created/updated.
• Tests added/updated, and test plan for scenarios not covered by automated tests.
• Equivalent MR/issue for omnibus-gitlab opened.
  • Expose circuit breaker values in workhorse config (gitlab-org/omnibus-gitlab#9276 - closed)

Reviewers checklist

• MR has a green pipeline on https://gitlab.com/gitlab-org/charts/gitlab.
• Consider downstream impact to the Operator, as per evaluating impact from changes to GitLab chart.