HTTP 5xx for TooManyIps error should be HTTP 403

Summary

While investigating an incident in GitLab Dedicated related to puma SLI violating the error rate SLO, it was discovered that when a user hits TooManyIps, all requests from this user will return HTTP 500.

Impact

Over the last 24 hours the SRE on-call was paged multiple times. The customer affected also opened an issue because no clear error message was presented.

Due to confidentiality, screenshots and logs are redacted and permalinks can't be provided.

Grafana webservice Service Overview dashboard:

Opensearch query filter and selected fields:

Results:

From the logs, extracted from the customer's S3 bucket, the underlying error has been found:

zgrep -h "User XXX from IP" logs/today/*/*|jq -r .message|sort -u
Gitlab::Auth::TooManyIps (User XXX from IP: 111.xxx.xxx.xxx tried logging from too many ips: 11):
Gitlab::Auth::TooManyIps (User XXX from IP: 222.xxx.xxx.xxx tried logging from too many ips: 11):
Gitlab::Auth::TooManyIps (User XXX from IP: 333.xxx.xxx.xxx tried logging from too many ips: 11):
Gitlab::Auth::TooManyIps (User XXX from IP: 444.xxx.xxx.xxx tried logging from too many ips: 11):

Recommendation

TooManyIps should return HTTP 403 and the web front end and API should return a clear error message (like "You/User XYZ tried logging in from too many IPs") and probably point to the correct configuration settings.

Verification

Logging in with one user from multiple public IPs should prove that the login will result in HTTP 500. In the production.log file the error Gitlab::Auth::TooManyIps (User XXX from IP: xxx.xxx.xxx.xxx tried logging from too many ips:) should be present.

Required settings: unique_ips_limit_enabled to be true and unique_ips_limit_per_user to be set to a reasonable value.