Skip to content

Circuitbreaker for storage paths

Bob Van Landuyt requested to merge bvl-nfs-circuitbreaker into master

What does this MR do?

In this MR, all access to the repository is wrapped in a a safeguard (CircuitBreaker). When an access to the repository fails because the the storage is not available (for example NFS not responding), this is logged. Subsequent requests will be blocked if they are too soon after the last failure, or if there have been too many failures, since that probably means that there is something wrong.

The amount of failures per storage are visible in the admin dashboard.

  • Circuitbreaker for storage access. And all access wrapped in it
  • Configuration of threshold for storage failures
  • Storage report in admin panel.
  • Reset failure counters from admin panel to unblock access.

Are there points in the code the reviewer needs to double check?

How do we define the thresholds for for blocking access to storage, currently hardcoded, but maybe it should be configured per storage in gitlab.yml.

Why was this MR needed?

This is needed to avoid GitLab marking repositories as missing.

https://gitlab.com/gitlab-org/gitlab-ce/issues/32207

Screenshots

GitLab blocking indefinitely:

Screen_Shot_2017-07-28_at_15.32.55

GitLab blocking temporarily:

Screen_Shot_2017-07-28_at_15.30.24

There was a recent failure, but the next attempt will be allowed:

Screen_Shot_2017-07-28_at_15.31.24

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

Closes #32207 (closed) Closes #33117 (closed) Closes https://gitlab.com/gitlab-com/infrastructure/issues/1946 Closes https://gitlab.com/gitlab-com/infrastructure/issues/1775

First step in https://gitlab.com/gitlab-org/gitlab-ce/issues/33220

Second attempt of: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10971

Omnibus Config changes: omnibus-gitlab!1749 (merged)

Edited by Bob Van Landuyt

Merge request reports