Maybe on other servers too, wi'll look into it on monday. ``` 2018-03-24 21:16:54 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="parse failed Empty input at line 1, column 1 [parse.c:926] in 'SCHED 214332569ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=220 runqueue=0 [0 0 0 0 0 0 0 0]" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin/filter_parser.rb:107:in `rescue in filter_with_time'" tag="gitaly" time=#<Fluent::EventTime:0x00007faa965f3050 @sec=1521926214, @nsec=662940000> record={"ident"=>"gitaly", "message"=>"SCHED 214332569ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=220 runqueue=0 [0 0 0 0 0 0 0 0]"} 2018-03-24 21:16:55 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="parse failed Empty input at line 1, column 1 [parse.c:926] in 'SCHED 214333575ms: gomaxprocs=8 idleprocs=7 threads=236 spinningthreads=0 idlethreads=220 runqueue=0 [0 0 0 0 0 0 0 0]" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin/filter_parser.rb:107:in `rescue in filter_with_time'" tag="gitaly" time=#<Fluent::EventTime:0x00007faad0c13a68 @sec=1521926215, @nsec=669740000> record={"ident"=>"gitaly", "message"=>"SCHED 214333575ms: gomaxprocs=8 idleprocs=7 threads=236 spinningthreads=0 idlethreads=220 runqueue=0 [0 0 0 0 0 0 0 0]"} 2018-03-24 21:16:56 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="parse failed Empty input at line 1, column 1 [parse.c:926] in 'SCHED 214334576ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=222 runqueue=0 [0 0 0 0 0 0 0 0]" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin/filter_parser.rb:107:in `rescue in filter_with_time'" tag="gitaly" time=#<Fluent::EventTime:0x00007faabbfea3b8 @sec=1521926216, @nsec=670590000> record={"ident"=>"gitaly", "message"=>"SCHED 214334576ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=222 runqueue=0 [0 0 0 0 0 0 0 0]"} 2018-03-24 21:16:57 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="parse failed Empty input at line 1, column 1 [parse.c:926] in 'SCHED 214335582ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=220 runqueue=0 [0 0 0 0 0 0 0 0]" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin/filter_parser.rb:107:in `rescue in filter_with_time'" tag="gitaly" time=#<Fluent::EventTime:0x00007faad0848f28 @sec=1521926217, @nsec=676350000> record={"ident"=>"gitaly", "message"=>"SCHED 214335582ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=220 runqueue=0 [0 0 0 0 0 0 0 0]"} 2018-03-24 21:16:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="parse failed Empty input at line 1, column 1 [parse.c:926] in 'SCHED 214336583ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=222 runqueue=0 [0 0 0 0 0 0 0 0]" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin/filter_parser.rb:107:in `rescue in filter_with_time'" tag="gitaly" time=#<Fluent::EventTime:0x00007faad0733f98 @sec=1521926218, @nsec=676960000> record={"ident"=>"gitaly", "message"=>"SCHED 214336583ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=222 runqueue=0 [0 0 0 0 0 0 0 0]"} 2018-03-24 21:16:59 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="parse failed Empty input at line 1, column 1 [parse.c:926] in 'SCHED 214337584ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=222 runqueue=0 [0 0 0 0 0 0 0 0]" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin/filter_parser.rb:107:in `rescue in filter_with_time'" tag="gitaly" time=#<Fluent::EventTime:0x00007faada3da9c8 @sec=1521926219, @nsec=678220000> record={"ident"=>"gitaly", "message"=>"SCHED 214337584ms: gomaxprocs=8 idleprocs=8 threads=236 spinningthreads=0 idlethreads=222 runqueue=0 [0 0 0 0 0 0 0 0]"} ``` /cc @gl-infra For now, I'll just clean up the tf-agent logs. Longterm, we need visibility and alerting into fluentd state which we never have time to do :disappointed: ``` $> be knife ssh 'roles:gitlab-base-stor-nfs' 'sudo du -sh /var/log/td-agent' | sort nfs-file-01.stor.gitlab.com 39M /var/log/td-agent nfs-file-02.stor.gitlab.com 19M /var/log/td-agent nfs-file-03.stor.gitlab.com 36M /var/log/td-agent nfs-file-04.stor.gitlab.com 20M /var/log/td-agent nfs-file-05.stor.gitlab.com 41M /var/log/td-agent nfs-file-06.stor.gitlab.com 49M /var/log/td-agent nfs-file-07.stor.gitlab.com 48M /var/log/td-agent nfs-file-08.stor.gitlab.com 59M /var/log/td-agent nfs-file-09.stor.gitlab.com 4.8G /var/log/td-agent nfs-file-10.stor.gitlab.com 3.9G /var/log/td-agent nfs-file-11.stor.gitlab.com 210M /var/log/td-agent nfs-file-12.stor.gitlab.com 198M /var/log/td-agent nfs-file-13.stor.gitlab.com 56M /var/log/td-agent nfs-file-14.stor.gitlab.com 125M /var/log/td-agent nfs-file-15.stor.gitlab.com 48M /var/log/td-agent nfs-file-16.stor.gitlab.com 317M /var/log/td-agent ```