Disable rate limiting on HAProxy
Because of:
- Rate limiting API feels broken on Gitlab.com: gitlab-org/gitlab#212753 (closed)
After:
- Migration of GitLab.com to Cloudflare: #1843 (closed)
Merge request
- Disable fe haproxy api rate limit enforcement: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2993
Change plan
Plan to drain each front-end haproxy lb node.
Change directory into gitlab-com-infrastructure/environments/gprd
.
For each node, run:
../../bin/drain_haproxy_node.sh fe-01-lb-gprd.c.gitlab-production.internal
Now, execute a thanos query for the relevant node fqdn and watch the node drain its connections.
Wait about 5 minutes or so. Then restart the given node by executing the following command:
ssh fe-01-lb-gprd.c.gitlab-production.internal 'sudo systemctl restart haproxy'
Check-list
-
fe-01-lb-gprd.c.gitlab-production.internal -
fe-02-lb-gprd.c.gitlab-production.internal -
fe-03-lb-gprd.c.gitlab-production.internal -
fe-04-lb-gprd.c.gitlab-production.internal -
fe-05-lb-gprd.c.gitlab-production.internal -
fe-06-lb-gprd.c.gitlab-production.internal -
fe-07-lb-gprd.c.gitlab-production.internal -
fe-08-lb-gprd.c.gitlab-production.internal -
fe-09-lb-gprd.c.gitlab-production.internal -
fe-10-lb-gprd.c.gitlab-production.internal -
fe-11-lb-gprd.c.gitlab-production.internal -
fe-12-lb-gprd.c.gitlab-production.internal -
fe-13-lb-gprd.c.gitlab-production.internal -
fe-14-lb-gprd.c.gitlab-production.internal -
fe-15-lb-gprd.c.gitlab-production.internal -
fe-16-lb-gprd.c.gitlab-production.internal -
fe-17-lb-gprd.c.gitlab-production.internal -
fe-18-lb-gprd.c.gitlab-production.internal -
fe-19-lb-gprd.c.gitlab-production.internal -
fe-20-lb-gprd.c.gitlab-production.internal -
fe-21-lb-gprd.c.gitlab-production.internal -
fe-22-lb-gprd.c.gitlab-production.internal -
fe-23-lb-gprd.c.gitlab-production.internal -
fe-24-lb-gprd.c.gitlab-production.internal
Edited by Nels Nelson