Skip to content

WIP: Set Filebeat to use daily indices

Zamir Martins requested to merge set_filebeat_to_daily_indices into master

What does this MR do?

After the most recent ELK version upgrade, indices changed from daily to a different heuristic as defined by ILM by default.

Related issue: #219643 (comment 359787092)

Current scenario:

  • GKE setup on 2020.06.09

  • On 2020.06.12 those are the available indices:

green open filebeat-7.6.2-2020.06.09-000001 Ku7WxTnwS1SprDBTiPfCYw 1 1 671424 0 1017.9mb 541.3mb
green open ilm-history-1-000001             z0vT1PgbRRSXaBX85UE2nQ 1 1     18 0   50.6kb  25.3kb

For any data range which doesn't not include 2020.06.09, the search (index based) will return empty.

for example:

While this one would accidentally return the 'right' results:

#input {"from":"2020-06-09T02:56:23.293Z","to":"2020-06-12T02:56:23.293Z"}
(from.to_date..to.to_date).map { |day| puts "filebeat-*-#{day.strftime('%Y.%m.%d')}-*"}
filebeat-*-2020.06.09-*
filebeat-*-2020.06.10-*
filebeat-*-2020.06.11-*
filebeat-*-2020.06.12-*

This one would return nothing (because 2020.06.09 contains all the data)

#input {"from":"2020-06-11T02:56:55.586Z","to":"2020-06-12T02:56:55.586Z"}
(from.to_date..to.to_date).map { |day| puts "filebeat-*-#{day.strftime('%Y.%m.%d')}-*"}
filebeat-*-2020.06.11-*
filebeat-*-2020.06.12-*

After additional testing, I will be closing this MR as the proposed changes still do not fix the issue. More investigation will be required.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Edited by Zamir Martins

Merge request reports