Modify Search::Zoekt::NamespaceIndexerWorker to schedule with random delay
Problem to solve
The Search::Zoekt::NamespaceIndexerWorker
currently indexes all projects into Zoekt without any delay. This has the potential to quickly grow the queue and schedule too many indexing jobs at once.
Proposal
- Use the already existing
MAX_JOBS_PER_HOUR
to add a random delay when scheduling all projects for a namespace. Mimic the pattern used in theGroups::TransferService
Comment below from @dgruzd
the actual ceiling for indexing is 16 concurrent jobs per node https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl#L28 We have 4 nodes, so 4 * 16 = 64 concurrent jobs total
-
MAX_JOBS_PER_HOUR
is likely set too high at 3600. Use the Kibana lens (internal) to determine how long jobs are taking to complete and updateMAX_JOBS_PER_HOUR
. We can initially use the last week of 2023 to get a concurrency limit, but should follow up in a month to adjust.
Edited by Terri Chu