Add gitlab shared-runners managers to whitelist

Production Change - Criticality 4 C4

Change Objective	Whitelist the shared-runners manager nodes so they don't get rate limit errors.
Change Type	Configuration change
Services Impacted	HAproxy
Change Team Members	@ggillies
Change Severity	C4 - Addition of IP to whitelist
Buddy check or tested in staging	A colleague will review the change or The change was tested on staging environment
Schedule of the change	6:55am Sept 23rd 2019 UTC
Duration of the change	30 minutes for complete chef rollout
Detailed steps for the change. Each step must include:

• Run apply_to_prod on pipeline (https://ops.gitlab.net/gitlab-cookbooks/chef-repo/pipelines/81643) for MR https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1868
• Run chef on fe-01-lb-gprd, verify haproxy picks up the change and doesn't complain
• Allow chef to run on the remaining LB nodes.

Rollback:

• Revert the MR, run apply_to_prod |

added C4 ~7170326 labels

changed the description

Copying from: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1868#note_34257

@ggillies a couple of questions, please:

1. Did we already test this in staging and did we see the expected outcome or does this issue not happen in staging?
2. By whitelisting, do we know how much traffic we would be letting in? (Any concern of opening a floodgate?)
3. According to a recent discussion around these runners in: #1146 (comment 218182943), my understanding is that these runners get torn down, re-created based on usage over time. I am not sure if the IPs will be the same once the current ones hit the limit before Auto-scaling takes them down and re-creates new ones via Docker Machine.

@aamarsanaa very good questions!

1. These have now been applied in staging and haproxy reloaded/worked ok with the new IPs added. As for testing if this resolves the issue, I think unfortunately it is hard for us to test in staging as staging doesn't get enought workloads to get rate limtied
2. My understanding is (and this could be wrong as I came in late to this issue and was following @cmiskell) is that this is more of re-opening a floodgate that used to be open. In other words, we are inadvertently rate limiting something which, until recently, wasn't rate limited. So applying this shouldn't flood us over what we had before
3. These are the runner-managers (or should be, unless I have done something very wrong), which are more "pet like" than the actual runners

actually in regards to 1) it seems this hasn't been applied to staging (I should have made one for gstg). I'm not sure of the impact here, as the whitelist for the gstg role seems very out of sync now with the gprd one (beyond just this MR).

changed the description

The context is a bit murky; for the record, we first got reports that CI reporting traces was getting rate-limited. I assumed that this was an implication of Cloud-NAT, because of the timing and nature of the issue, leading to #1178 (closed). However that did not have the required effect. I used BigQuery, ultimately

SELECT count(*) , jsonPayload.c_ip FROM `gitlab-production.haproxy_logs.haproxy_20190923` WHERE jsonPayload.status_code='429' AND jsonPayload.path LIKE '%/trace' GROUP BY jsonPayload.c_ip

which showed the IPs being rate-limited were actually the runner managers, which actually makes more sense (given it's 'traces' being uploaded).

This does not explain why this started happening around the time that the cloud NAT work happened, as the IPs and network path for the runner managers shouldn't (AFAIK) have changed. However, they are being rate-limited, they are valid, so IMO we should un-rate-limit them and see how that goes. We still need to understand the cause of the change in behavior, but we making things work cleanly is the first priority.

None of this will apply to staging, because the volume just isn't there.

https://dashboards.gitlab.net/d/AkOdlrSmk/imported-haproxy-stats?orgId=1&refresh=5m&from=now-7d&to=now&fullscreen&panelId=5 may also be useful; it shows the rate of 429's, and the base load of those went up around 2019-09-20 00:00, which is co-incidental timing with the some changes related to cloud NAT (I think that was when @craigf applied https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1061

That timing is correct @cmiskell. It's strange that its the managers that are getting rate limited. Cloud NAT only NATs packets from instances without a public IP. ssh'ing onto a runner manager and running curl https://api.ipify.org verifies that the apparent NAT IP is equal to the public IP of the instance, not a Cloud NAT IP.

Looking that the chart, I don't see any increase in 429s around that time. There is a spike shortly after, but there have been periodic spikes roughly every 24h.

changed the description

marked the checklist item Run apply_to_prod on pipeline (https://ops.gitlab.net/gitlab-cookbooks/chef-repo/pipelines/81643) for MR https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1868 as completed

marked the checklist item Allow chef to run on the remaining LB nodes. as completed

marked the checklist item Run chef on fe-01-lb-gprd, verify haproxy picks up the change and doesn't complain as completed

Applied

closed

mentioned in issue #1182 (closed)

mentioned in issue #1184 (closed)