2017-09-04 -Logstash rootfs out of space

At 01:17 AM UTC, a PagerDuty alert fired for logstash's root fs being full. Upon investigation this was because there was a 19GB log file /var/log/logstash/logstash-plain.log. I copied this file to /mnt (the temp disk from Azure). As soon as I moved that file out of the way, a new one began and started to fill up very fast, a sample of which is below.

:response=>{"index"=>{"_index"=>"logstash-2017.09.05", "_type"=>"haproxy", "_id"=>"AV5PpBypRc265xQb7I6w", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [syslog_timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"Sep  5 01:16:17\""}}}}}

:response=>{"index"=>{"_index"=>"logstash-2017.09.05", "_type"=>"linux-auth", "_id"=>"AV5PpBypRc265xQb7I6j", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [syslog_timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"Sep  5 01:02:48\""}}}}}

cc/ @gl-infra @briann