Fix dropped logs in the new Elasticsearch 7 cluster
Background
ES5 has always failed to index certain logs. We do not want this to remain the case in ES7, as much as possible. Due to differing strictness in schema enforcement, the sets of dropped logs in each ES version are disjoint. We are focussing our efforts on the new cluster, as fixing ES5 errors is likely to be wasted effort after the migration is complete.
first example:
this log entry is only in log.gitlab.net: https://log.gitlab.net/goto/18ef3924c13fae72e0ae11563d7c14ea / https://log.gprd.gitlab.net/goto/3f4ecceb8fe86ee7b5cf24c1ab8db6d9
{
"query": {
"match": {
"json.jid.keyword": {
"query": "e3d89cfd414c18370c9dbf56",
"type": "phrase"
}
}
}
}
Edited by Craig Furman