Add monitoring for Elasticsearch index shard sizes
Background
The main Elasticsearch index gitlab-production
has 120 shards with shard sizes that on average are around 40GB
. The Elastic docs recommend to aim to keep shard sizes below 50GB
.
As content continues to grow, we will start to have shards that are too big.
On the other hand, shard sizes in gitlab-notes
are too small. They are around 130.5MB
earch.
Proposal
We should add monitoring and alerting to alert us if shard sizes are outside of a specified range and require us to do a reindex.