fluentd: emit unmatched (non-json) lines
I was looking at the fluentd logs (/var/log/td-agent/td-agent.log
) on one of the pages hosts (web-pages-03-sv-gprd.c.gitlab-production.internal
) and noticed that we have a bunch of non-JSON lines being dropped:
2020-02-21 10:48:56 +0000 [warn]: #0 pattern not matched: "2020/02/21 10:48:56 http: TLS handshake error from 10.216.2.3:37978: EOF"
This is likely affecting other hosts as well. While there doesn't appear to be a prometheus metric for this, there is a setting called emit_unmatched_lines
.
This setting will construct a JSON record for those lines that looks like this:
{"unmatched_line":"2020-02-21 10:48:56 +0000 [warn]: #0 pattern not matched: \"2020/02/21 10:48:56 http: TLS handshake error from 10.216.2.3:37978: EOF\""}
Which might spam kibana a bit, but at least gives us better insight into what is being dropped -- and allows searches on those non-structured logs.
The goal should be to migrate those to JSON -- that's something we can then talk to the dev teams about.
@craigf @bjk-gitlab @aamarsanaa @hphilipps @craig @cindy @jarv