Blacklist URL at HAProxy Frontend
C3
Production Change - Criticality 3Change Objective | Block a URL identified by SecOps as being hit aggressively to serve a 1-byte file as a probe point. |
---|---|
Change Type | ConfigurationChange |
Services Impacted | Frontend HAProxy fleet (Chef role gprd-base-lb-fe-common , hostname pattern fe-XX-lb-gprd ) |
Change Team Members | @msmiley |
Change Severity | C3 |
Change Reviewer or tested in staging | @nnelson |
Dry-run output | N/A, this is a chef-client run. |
Due Date | 2019-11-20 19:15 UTC (11:15 PDT) |
Time tracking | 30 minutes to apply to 24 nodes (and equal time to rollback) |
Preconditions
Merged the merge-request that adds the blacklist entry:
https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2193
Apply procedure
Run the apply_to_prod
pipeline for the above merge-request, to apply the role changes to the Chef server.
Serially run chef-client
on all 24 frontend HAProxy nodes.
$ knife ssh -p 2222 -C1 'roles:gprd-base-lb-fe-common' 'sudo chef-client'
Monitor the GitLab Triage dashboard:
https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage
Rollback procedure
Revert the merge request, and run chef-client
on any hosts where it had already been applied.
https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2193
It is strongly recommended to:
- Note relevant graphs in grafana to monitor the effect of the change, including how to identify that it has worked, or has caused undue negative effects
- Review alerts that may go off that can be silenced pro-actively