Mark JobWaiter jobs with context metadata
Our JobWaiter class allows a Sidekiq job to receive an optional second argument, which is a waiter key. Another process (typically a controller action) can use that key to wait for a set of Sidekiq jobs to complete (with a sensible timeout).
In #86 (closed) and #42 (comment 266872492), we can see that the majority of authorized_projects jobs use this second argument. However, it's quite fiddly to do that because we need to filter on the presence of a second argument, which isn't trivial with an Elasticsearch array.
Instead, I think we could make the JobWaiter jobs have some context metadata indicating that they are waited-on, which allows us to filter by that in Kibana. It does not let us define separate SLOs for them, or filter them separately in Prometheus, but I think that's OK as we probably have too many Sidekiq queues right now (https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues - 'I don't recommend having more than a handful of queues').
Previous proposal
Instead, I think we could make the JobWaiter jobs use their own namespaced queue. So we'd have:
- authorized_projects with an argument of
123
.- authorized_projects:job_waiter with arguments of
123
andjobwaiterkey
.That means that we could easily filter between the two queues, and potentially even apply different SLOs to them. We could also record metrics separately (this particularly interesting for enqueue rate: #91 (closed)).
Because of the way queue namespaces work, this shouldn't require any infrastructure changes: anything processing authorized_projects should also process authorized_projects:job_waiter.