Skip to content

Add pack-objects concurrency limiter based on RemoteIP

ThisMR adds one more concurrency limiting strategy for pack-objects. This strategy depends on RemoteIP passed from the downstream services. As the foundation for concurrency limiting is solid, the change is minimal. This commit just needs to declares a key before triggering runPackObjectsLimited method. Other observability stuff are included automatically like magic.

The configurations for pack-objects limiting is fed from two environment variables:

  • GITALY_PACK_OBJECTS_LIMIT_MAX_CONCURRENCY
  • GITALY_PACK_OBJECTS_LIMIT_MAX_QUEUE_LENGTH

Setting the config this way helps to boost the integration speed significantly. If we go through a typical Omnibus/K8s integration, we will have to wait for a long time before seeing the results. Furthermore, those configs may differ between nodes. It takes some iterations before we come up with a reasonable number. Therefore, I would like to follow this approach. When the new limiter becomes stable, I'll add supports for those configs to the distribution platform in a proper way.

Production configuration

The configuration in production is calculated from the concurrency IP logging. To sum up, I target to limit the concurrency per IP per node to 15. Requests exceeding this limit are queued. The maximum queue length is 200, maximum queue time is 5 minutes. More about those numbers are analyzed here: https://gitlab.com/gitlab-org/gitlab/-/issues/396294#note_1346383032.

Edited by Quang-Minh Nguyen

Merge request reports