We see high CPU utilization on instance-0000000022 from a few hours ago.
The indexing threads seem to also be growing for this node. No issues on other nodes we can see.
We also see the elastic_commit_indexer
queue growing. It's not clear if these are connected.
![Screen_Shot_2020-06-23_at_9.57.15_am](data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==)
Elastic Support Case
https://support.elastic.co/customers/s/case/5004M00000dDbhhQAC
Error logs from indexing workers
https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))&_a=(columns:!(json.error_class,json.error_message),filters:!(('state':(store:appState),meta:(alias:!n,disabled:!f,index:AW5F1e45qthdGjPJueGO,key:json.class.keyword,negate:!f,params:!(ElasticIndexBulkCronWorker,ElasticCommitIndexerWorker,ElasticIndexInitialBulkCronWorker),type:phrases,value:'ElasticIndexBulkCronWorker,%20ElasticCommitIndexerWorker,%20ElasticIndexInitialBulkCronWorker'),query:(bool:(minimum_should_match:1,should:!((match_phrase:(json.class.keyword:ElasticIndexBulkCronWorker)),(match_phrase:(json.class.keyword:ElasticCommitIndexerWorker)),(match_phrase:(json.class.keyword:ElasticIndexInitialBulkCronWorker)))))),('
state':(store:appState),exists:(field:json.error_backtrace),meta:(alias:!n,disabled:!f,index:AWNABDRwNDuQHTm2tH6l,key:json.error_backtrace,negate:!f,type:exists,value:exists))),index:AWNABDRwNDuQHTm2tH6l,interval:auto,query:(language:kuery,query:''),sort:!(!(json.time,desc)))