Circuitbreaker for storage paths
What does this MR do?
In this MR, all access to the repository is wrapped in a a safeguard (CircuitBreaker). When an access to the repository fails because the the storage is not available (for example NFS not responding), this is logged. Subsequent requests will be blocked if they are too soon after the last failure, or if there have been too many failures, since that probably means that there is something wrong.
The amount of failures per storage are visible in the admin dashboard.
-
Circuitbreaker for storage access. And all access wrapped in it -
Configuration of threshold for storage failures -
Storage report in admin panel. -
Reset failure counters from admin panel to unblock access.
Are there points in the code the reviewer needs to double check?
How do we define the thresholds for for blocking access to storage, currently hardcoded, but maybe it should be configured per storage in gitlab.yml.
Why was this MR needed?
This is needed to avoid GitLab marking repositories as missing.
https://gitlab.com/gitlab-org/gitlab-ce/issues/32207
Screenshots
GitLab blocking indefinitely:
GitLab blocking temporarily:
There was a recent failure, but the next attempt will be allowed:
Does this MR meet the acceptance criteria?
-
Changelog entry added, if necessary -
API support added - Tests
-
Added for this feature/bug -
All builds are passing
-
-
Squashed related commits together
What are the relevant issue numbers?
Closes #32207 (closed) Closes #33117 (closed) Closes https://gitlab.com/gitlab-com/infrastructure/issues/1946 Closes https://gitlab.com/gitlab-com/infrastructure/issues/1775
First step in https://gitlab.com/gitlab-org/gitlab-ce/issues/33220
Second attempt of: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10971
Omnibus Config changes: omnibus-gitlab!1749 (merged)