Disable VPC flow logs
This spins out of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6343 which is concerned about reducing our Stackdriver costs, which are very high. VPC flow logs are the main component of these costs.
VPC flow logging was requested by the security team last year (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4701) and implemented in August 2018.
This is costing us about ~$5000 a week or $20k per month, roughly half of our total Stackdriver Logging spend. The storage costs for these logs add even more to the total cost.
This graph shows the breakdown in costs in Stackdriver:
https://docs.google.com/spreadsheets/d/1nMpeAmeAyWFwQsfZW4ICmch0I1mL7iMAzkEnForeHNU/edit?usp=sharing
These graphs are at low level, very verbose, and as far as I can tell, have never been used in a security investigation or for other purposes. If this is indeed the case (which I may be wrong about of course), then I think we should consider turning them off until we have a strategy in place for working with them.
Here is a typical example of one of these logs:
{
jsonPayload: {
bytes_sent: "126"
connection: {
dest_ip: "10.220.4.5"
dest_port: 37170
protocol: 6
src_ip: "10.221.2.118"
src_port: 9999
}
dest_instance: {
project_id: "gitlab-production"
region: "us-east1"
vm_name: "git-10-sv-gprd"
zone: "us-east1-c"
}
dest_vpc: {
project_id: "gitlab-production"
subnetwork_name: "git-gprd"
vpc_name: "gprd"
}
end_time: "2019-04-18T10:24:07.695920491Z"
packets_sent: "10"
reporter: "SRC"
rtt_msec: "0"
src_instance: {
project_id: "gitlab-production"
region: "us-east1"
vm_name: "file-18-stor-gprd"
zone: "us-east1-c"
}
src_vpc: {
project_id: "gitlab-production"
subnetwork_name: "file-gprd"
vpc_name: "gprd"
}
start_time: "2019-04-18T10:24:07.551160806Z"
}
logName: "projects/gitlab-production/logs/compute.googleapis.com%2Fvpc_flows"
receiveTimestamp: "2019-04-18T10:24:13.283226613Z"
resource: {
labels: {
location: "us-east1-c"
project_id: "gitlab-production"
subnetwork_id: "592989008458390188"
subnetwork_name: "file-gprd"
}
type: "gce_subnetwork"
}
timestamp: "2019-04-18T10:24:13.283226613Z"
}
In addition to the Stackdriver costs, these VPC logs are accruing a huge quantity of data in Google Cloud Storage - 1.7 Terabytes a day!. In March, we stored 47TB of these logs.
Considering the logs don't appear to have any retention policy, and each months logs cost us over $1300 per month, which is additive (ie, each month the amount we spend on storing these logs will go up by $1300) I think we should shut this down.