GCP firewall rule and forwarding rule quota - gitlab-review-apps
Problem
The gitlab-review-apps
project firewall rule quota is near the limit and there may be rules that are orphaned or not understood. Original discussion is at https://gitlab.slack.com/archives/CMA7DQJRX/p1584410955103800
@kwiebers @rymai - I was looking into creating another issue to bump the firewall rule quota similar to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/6642 but am misunderstanding the number of firewall rules per release. As of writing there seems to be 590 firewall rules in use by gitlab-review-apps. From what I can tell in GCP we need 6 firewall rules per release: 2 for nginx-ingress-controller-svc (443,22) https://console.cloud.google.com/kubernetes/service/us-central1-b/review-apps-ee/review-apps-ee/review-11024-add-zs5uux-nginx-ingress-controller?project=gitlab-review-apps&tab=overview&duration=PT1H&pod_summary_list_tablesize=20 1 for minio-svc via K8s ingress (9000) https://console.cloud.google.com/kubernetes/service/us-central1-b/review-apps-ee/review-apps-ee/review-11024-add-zs5uux-minio-svc?project=gitlab-review-apps&tab=overview&duration=PT1H&pod_summary_list_tablesize=20 1 for registry-svc via K8s ingress (5000) https://console.cloud.google.com/kubernetes/service/us-central1-b/review-apps-ee/review-apps-ee/review-11024-add-zs5uux-registry?project=gitlab-review-apps&tab=overview&duration=PT1H&pod_summary_list_tablesize=20 2 for unicorn-svc via K8s Ingress (8080, 8181) https://console.cloud.google.com/kubernetes/service/us-central1-b/review-apps-ee/review-apps-ee/review-11024-add-zs5uux-unicorn?project=gitlab-review-apps&tab=overview&duration=PT1H&pod_summary_list_tablesize=20 There are 42 releases via helm ls -d | wc -l which adds up to 252 that is way less than the current 590. Do you see anything that I’m missing?
@rymai I think you’re right and I actually noticed the same yesterday: that the total number of rules we have is far greater than the (number of Review Apps) * (rules per RA). It seems there are rules like https://console.cloud.google.com/networking/firewalls/details/k8s-a00947bb4649011ea9f9f42010af0022-http-hc?project=gitlab-review-apps&authuser=0 with a random port, it seems. There are approx. 286 of those…
@rymai I think these are the controller.service.healthCheckNodePort as documented in https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/charts%2Fnginx.
If controller.service.type is NodePort or LoadBalancer and controller.service.externalTrafficPolicy is set to Local, set this to the managed health-check port the kube-proxy will expose. If blank, a random port in the NodePort range will be assigned
Questions
-
How many rules are needed per review app instance?What resources are needed per review app instance? -
Are there unattached rules that remain?
Yes. As of 2020-03-23T05:08:10+00:00, there are 181 rules in review-apps-ee
, but there are only 6 nginx-ingress-controller
service in review-apps-ee
-
Why are there random ports opened for certain rules like https://console.cloud.google.com/networking/firewalls/details/k8s-a00947bb4649011ea9f9f42010af0022-http-hc?project=gitlab-review-apps&authuser=0?
The ports that are not 80, 443, 80 are used by nginx-ingress-controller service's HealthCheckNodePorts. These are automatically created by GKE control plane when the nginx-ingress-controller's service type is LoadBalancer and externalTrafficPolicy is "Local"
-
Is there cleanup required for these rules or should they be automatically cleaned up as apart of the environment being removed?
These should be cleaned up by GKE control plane, but seems like there was a bug recently. Filed https://issuetracker.google.com/issues/152137011 to GCP for investigation. If manual clean up was required, we would have seen many more dangling resources, given the number of review app deployments happening daily.
What does success look like?
- Understand more about firewall rules that are being created per release
- Delete any orphaned firewall rules
- Ensure consistency in ports for application purposes in a release