[Feature flag] Enable pack objects limiting for cny

What

Enable the :gitaly_pack_objects_limiting_repo OR :gitaly_pack_objects_limiting_user OR :gitaly_pack_objects_limiting_remote_ip feature flags.

This is to test out what is the more effective strategy--to limit pack object concurrency by user or by repo. The configurations are fed from the environment variables. By default, the max concurrency is set to 200.

• GITALY_PACK_OBJECTS_LIMIT_MAX_CONCURRENCY
• GITALY_PACK_OBJECTS_LIMIT_MAX_QUEUE_LENGTH

#4401 (closed) contains some graphs for how much concurrency we see per user or per repository. Based on #4401 (comment 1084920115), it sounds like we may want to start with limiting per repository since that's what cgroups as well as the concurrency limiter does, so to add the ability to limit per user might be hard to reason about from a systems perspective.

Owners

• Team: Gitaly
• Most appropriate slack channel to reach out to: #g_create_gitaly
• Best individual to reach out to: @qmnguyen0711

Expectations

What release does this feature occur in first?

15.11

What are we expecting to happen?

When users excessively trigger pack objects processes, their clones will be delayed

What might happen if this goes wrong?

Too many users will have their clones delayed

What can we monitor to detect problems with this?

The following metrics should be monitored:

gitaly_pack_objects_acquiring_seconds

gitaly_pack_objects_in_progress

gitaly_pack_objects_queued

gitaly_pack_objects_dropped_total

Roll Out Steps

• Enable on staging
  • Is the required code deployed on staging? (howto)
  • Enable on staging (howto)
  • Add featureflagstaging to this issue (howto)
  • Test on staging (howto)
  • Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
• Enable on production
  • Is the required code deployed on production? (howto)
  • Enable on production in #production (howto)
  • Add featureflagproduction to this issue
  • Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
• Default-enable the feature flag (optional, only required if backwards-compatibility concerns exist)
  • Wait for release containg default-disabled feature flag.
  • Change the feature flag to default-enabled (howto)
  • Wait for release containing default-enabled feature flag.
• Remove feature flag
  • Remove the feature flag and the pre-feature-flag code (howto)
  • Remove the feature flag via chatops (howto)
  • Close this issue

Please refer to the documentation of feature flags for further information.