[Feature flag] Enable use of concurrency queue limits
What
Enable the :gitaly_concurrency_queue_max_wait
feature flag and :gitaly_concurrency_queue_enforce_max
. These feature flags enable Gitaly to put a limit on how large the concurrency queue can grow.
Owners
- Team: Gitaly
- Most appropriate slack channel to reach out to:
#g_create_gitaly
- Best individual to reach out to: jcaigitlab
Expectations
What release does this feature occur in first?
What are we expecting to happen?
When the concurrency queue for an RPC bursts the limit set, we should see "maximum queue size reached" errors, or "maximum time in concurrency queue reached" errors.
What might happen if this goes wrong?
Requests fail with the above error messages even when the concurrency limits have not reached the configured limits.
dashboard:
What can we monitor to detect problems with this?
- [Concurrency Queue Status](https://dashboards.gitlab.net/d/CP23bQfnz/gitaly-concurrency-queue?orgId=1
Roll Out Steps
-
Enable on staging-ref -
Enable on staging -
Is the required code deployed on staging? (howto) -
Enable on staging (howto) -
Add featureflagstaging to this issue (howto) -
Test on staging (howto) -
Verify the feature flag was used by checking Prometheus metric [ gitaly_feature_flag_checks_total
]
-
-
Enable on production -
Is the required code deployed on production? (howto) -
Enable on production in #production
(howto) -
Add featureflagproduction to this issue -
Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
-
-
Default-enable the feature flag (optional, only required if backwards-compatibility concerns exist) -
Wait for release containg default-disabled feature flag. -
Change the feature flag to default-enabled (howto) -
Wait for release containing default-enabled feature flag.
-
-
Remove feature flag
Please refer to the documentation of feature flags for further information.
Edited by John Cai