Consider dropping unstructured rails logs
In https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12928, @stanhu added structured logging for rails processes.
Although this change introduced structured logging, it kept the original unstructured logging too. This gave us the ability to test the new log structure alongside the original log format.
On GitLab.com, we have now transitioned completed to using the structured logs and rely heavily on them, whereas the unstructured access logs are also ingested but not used for analysis.
Unfortunately, these unstructured logs are the second largest source of log data - almost 300GB per day, even though we don't use them.
We could stop ingesting unstructured logs altogether, but there is other useful information mixed in with these logs too, and which we would like to keep.
config.lograge.keep_original_rails_log = false
Would stop duplicating the unstructured access logs alongside the structured logs. At the very least, we should make this an option, which we can turn off for GitLab.com.