Extreme memory usage during initial indexing

I am having issues enabling indexing on an omnibus Gitlab instance, of which the main symptom is OOM including swap.


GitLab instance details:

GitLab 13.9.1-ee (8ae438629fa) Omnibus
Ubuntu 16.04 on AWS EC2

Indexer configuration:

shards = 1
replicas = 0
max indexed file size = 10000 KiB
field length = 0
bulk request size = 30 MiB
bulk request concurrency = 1
client request timeout = 60

ElasticSearch:

Single-node on Kubernetes, v7.11.1


Our GitLab instance normally runs in an m4.large instance type without issues. Currently configured memory is 8GB RAM and 5GB swap. Backup size excluding registry is ~ 27GB. When indexing is started, htop shows indexer threads very quickly consuming RAM and eventually filling swap, causing OOM killer to be invoked, and rendering the instance inoperable. This occurs on the order of 5-10 minutes or less.

Today, I attempted to remedy this by swapping the instance type to m4.xlarge (4CPU, 16GB + 5GB) and memory filled up even faster. A swap to r4.xlarge (4CPU, 30GB + 5GB) resulted in filling up all memory and swap to invoke OOM killer in less than 3 minutes.

image

Based on the information above, is there anything I can test to see if this is a bug, or is this user error? Elasticsearch has not been stressed during this ordeal and returns searches based on keywords in 1s or less. Its index currently contains 109K documents in ~300mb (which is probably a quarter or less of all projects, in my estimation).