Tracking state of mod security on version.gitlab.com for WAF Troubleshooting
The defend team is trying to reproduce the problem we are seeing with mod_security
on the kubernetes ingress controllers. gitlab-org/gitlab#39140 (closed) - After the troubleshooting meeting this afternoon (Recorded), it was clear that there was not enough time in the meeting to see the problem happening. (meeting notes here: https://docs.google.com/document/d/1OQWxl3ARwojQFd0SIgd5VlNiG1khR0xz8ANz9KJmL1M/edit#heading=h.6ozy7jtc8hcn)
In an effort to help with their troubleshooting, we have re-enabled mod_security
on version.gitlab.com. This issue is to track this activity, and should be closed when mod_security
is disabled, or when the root cause is discovered. The actual troubleshooting should be done on the issue above.
Alerts and Monitoring
The status of this service can be seen on the following graphs: https://dashboards.gitlab.net/d/Eo8BHoNZz/version-gitlab-com?orgId=1&refresh=5m&from=1578343909067&to=1579121509067
If this starts showing intermittent failures again, notify @plafoucriere
If The on-call starts getting alerts regarding version.gitlab.com being unresponsive, then mod_security
should be disabled.
Disabling
- Get credentials and set the cluster context:
gcloud container clusters get-credentials gs-production-gke --region us-east1 --project gs-production-efd5e8
- Edit the ingress config map:
kubectl -n gitlab-managed-apps edit configmap ingress-nginx-ingress-controller
- Set
enable-modsecurity: "false"
andenable-owasp-modsecurity-crs: "false"
- Save and exit
- Close this issue