Infra investigation followup: Malformed log entries in kibana
Summary
Are logs being malformed? What percentage of our logs are malformed and can we prevent this from happening more often?
Example of malformed logs: https://log.gprd.gitlab.net/app/discover#/doc/7092c4e2-4eb5-46f2-8305-a7da2edad090/pubsub-rails-inf-gprd-014551?id=hmBl7oQBR8eYS4p63kyv
This issue is possibly made more obvious since we are dropping logs in pubsubbeat with the subcomponent of database_load_balancing
. It may be helpful to re-enable these logs and look for problems. A current hypothesis is that dropping these logs allowed us to see malformed logs that were there before. The malformation possibly being the cause of why these logs were not dropped properly.
To minimize the increase in logging, another application is here that would help offset the log volume.
Related Incident(s)
Originating issue(s): production#8140 (closed)