Move ReactiveCachingWorker priority to low
What does this MR do?
Now that we moved reactive_caching queue work that perform external requests to a separated worker with external dependency and low priority, now it's time to tune down the reactive_caching queue to low priority (but keep it CPU-bound). But why?
Currently it just performs caching work for MergeRequest (see graphs at gitlab-com/gl-infra/scalability#224 (comment 333796982)), and these just perform CPU-bound work for CI reports (see gitlab-com/gl-infra/scalability#224 (closed)). This work can easily take more than 10 seconds of execution (our execution SLO).
According to our docs:
If a worker cannot meet these expectations, then it cannot be treated as a urgency :high worker: consider redesigning the worker, or splitting the work between two different workers, one with urgency :high code that executes quickly, and the other with urgency :low, which has no execution latency requirements (but also has lower scheduling targets).
We've splitten the external dependency work, but the remaining MergeRequest related work in this queue is still quite expensive to be high priority (with a p99 of ~100 seconds).
Part of the work for gitlab-com/gl-infra/scalability#224 (closed).