Combine timestamp and log id when filtering slowlog entries
When redis restarts, the slowlog gets reset. In our slowlog plugin we filter slowlog entries we've already emitted using the slowlog id. This means that if redis gets restarted while td-agent is running, we'd miss slowlog entries until the entry id is higher than the last one that was emitted
To work around this, we could take the timestamp into account: if we store an identifier based on `timestamp + id`, then all newer log entries would be emitted, regardless of their id. While we would still avoid emitting duplicate entries that happened within the same second.
To avoid duplicate log entries when td-agent restarts, the initial identifier should be `current_timestamp`, meaning we won't re-emit the entire slowlog.
____
1. https://gitlab.com/gitlab-org/fluent-plugin-redis-slowlog/-/blob/dd28a5e3608bf0bf94870f1c92a51fdb91571f18/lib/fluent/plugin/in_redis_slowlog.rb#L82 means it will always ignore the oldest slowlog entry because it has id 0, and current_log_id is 0 at startup.
1. When redis itself restarts the slowlog will be emptied and id's start from 0 again,
1. Conversely, when fluentd is restarted current_log_id starts from 0 again so all current slowlogs will be re-emitted.
Perhaps we should use the timestamp of the slowlog entries instead of just the ID, and *store* the last-seen timestamp (in the monitored redis, for simplicity) to survive fluentd restarts.
/cc @reprazent WDYT?
issue