Skip to content

Reduce the sensitivity of the unauthenticated connections rate limit

Summary

We recently expired all tokens with no expiration dates: https://about.gitlab.com/blog/2023/10/25/access-token-lifetime-limits/

During this activity, many users experienced IP blocks due to retrying authentication too frequently.

As part of temporary mitigation, via gitlab-com/gl-infra/k8s-workloads/gitlab-com!3590 (merged) we temporarily changed the values to:

 maxretry: 30 -> 500
 findtime: 180 -> 60
 bantime: 3600 -> 900

Follow up actions

We need to consider what impact this temporary change has had on the platform as a whole. We should then consider:

  1. Keeping the limit if our research shows that we are able to absorb the additional unauthenticated traffic
  2. OR adapt a more appropriate limit. Previous limit was too low and was causing various edge issues, so we need to find what a more appropriate limit should be.

This needs to be completed by 2024-05-24.

Related Incident(s)

Originating issue(s): production#18003 (closed)

Associated Services

ServiceRateLimiting

Corrective Action Issue Checklist

  • Link the incident(s) this corrective action arose from
  • Give context for what problem this corrective action is trying to prevent re-occurring
  • Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
  • Assign a priority (this will default to 'Reliability::P4' but should match the severity of the related incident)
  • Assign a service label
Edited by Marin Jankovski