2020-06-23: High CPU on Elasticsearch Advanced Global Search nodes

We see high CPU utilization on instance-0000000022 from a few hours ago.

The indexing threads seem to also be growing for this node. No issues on other nodes we can see.

We also see the elastic_commit_indexer queue growing. It's not clear if these are connected.

Elastic Support Case

https://support.elastic.co/customers/s/case/5004M00000dDbhhQAC

Error logs from indexing workers

https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))&_a=(columns:!(json.error_class,json.error_message),filters:!(('state':(store:appState),meta:(alias:!n,disabled:!f,index:AW5F1e45qthdGjPJueGO,key:json.class.keyword,negate:!f,params:!(ElasticIndexBulkCronWorker,ElasticCommitIndexerWorker,ElasticIndexInitialBulkCronWorker),type:phrases,value:'ElasticIndexBulkCronWorker,%20ElasticCommitIndexerWorker,%20ElasticIndexInitialBulkCronWorker'),query:(bool:(minimum_should_match:1,should:!((match_phrase:(json.class.keyword:ElasticIndexBulkCronWorker)),(match_phrase:(json.class.keyword:ElasticCommitIndexerWorker)),(match_phrase:(json.class.keyword:ElasticIndexInitialBulkCronWorker)))))),('state':(store:appState),exists:(field:json.error_backtrace),meta:(alias:!n,disabled:!f,index:AWNABDRwNDuQHTm2tH6l,key:json.error_backtrace,negate:!f,type:exists,value:exists))),index:AWNABDRwNDuQHTm2tH6l,interval:auto,query:(language:kuery,query:''),sort:!(!(json.time,desc)))