Gitaly Log Volumes

As part of the GCP Migration work in gitlab-com/migration#355 (closed), we now have some interesting graphs which show us log volumes for each component sending to the Elastic cluster.

https://app.google.stackdriver.com/monitoring/1088234/logging-pubsub-in-gprd?project=gitlab-production

One take-away is that Gitaly is far and away the greatest generator of logs on GitLab.com at present.

image

Should we start considering selectively multiple loggers or other solutions for tuning this down?

One easy win might be to drop the spawn log, which, in the last 15 minutes generated 1.5M log entries alone but which doesn't add much to our observability:

image

cc @jarv