Logstash stops sending some logs to ES
It seems after some hours parsing and sending logs to ES, logstash
stops keeping track of "newer" logfiles (we use YYYYMMDDHH
format).
So on Kibana, there will be blank spots for some of the log types.
It seems that sometimes the newer logs (from the very same hour) are not tracked by logstash:
root@logstash:~# lsof -p 30191 | grep gitlab-log | grep 52.225.131.122
java 30191 logstash 110r REG 252,0 183327419 598498833 /gitlab-log/52.225.131.122/52.225.131.122-2017072010.gitlab.log
java 30191 logstash 113r REG 252,0 22183555 610768388 /gitlab-log/52.225.131.122/52.225.131.122-2017072013.auth.log
java 30191 logstash 277r REG 252,0 7365869 610768403 /gitlab-log/52.225.131.122/52.225.131.122-2017072014.auth.log
java 30191 logstash 293r REG 252,0 70210486 610768393 /gitlab-log/52.225.131.122/52.225.131.122-2017072013.audit.log
java 30191 logstash 436r REG 252,0 287157 610768389 /gitlab-log/52.225.131.122/52.225.131.122-2017072013
java 30191 logstash 571r REG 252,0 3672 610768408 /gitlab-log/52.225.131.122/52.225.131.122-2017072014.mail.log
java 30191 logstash 623r REG 252,0 143415 610768407 /gitlab-log/52.225.131.122/52.225.131.122-2017072014
java 30191 logstash 692r REG 252,0 23626763 610768411 /gitlab-log/52.225.131.122/52.225.131.122-2017072014.audit.log
java 30191 logstash 882r REG 252,0 180598807 598498850 /gitlab-log/52.225.131.122/52.225.131.122-2017072011.gitlab.log
java 30191 logstash 920r REG 252,0 181032844 598498866 /gitlab-log/52.225.131.122/52.225.131.122-2017072012.gitlab.log
We need to see how we can fix this, as a restart
helps with the problem but it's obviously the last thing we want to do.