Add monitoring and alerts for embeddings indexer
Background
Embeddings were added to the index in #451431 (closed)
Proposal
- Add prometheus metrics to expose the queue size for the embeddings cron worker
The metrics are emitted in the Elastic::MetricsUpdateService
- Consume new metrics and generate alerts when the queue is backed up.
The existing alerts are defined in https://gitlab.com/gitlab-com/runbooks and for production and staging. The alerts are fired when the queue reaches 10_000 for staging and 50_000 for production