Define arbitrary queue attributes using tag sets
This is a fairly speculative proposal. I'm not totally sold on it, but present it as a proposal as I see several similar efforts and feel that if we were to carry them all out, implementing them all with this would be better than multiple alternatives.
The Problem
Currently we have a several proposals to add additional attributes for workers.
These include: #139 (closed) and #106 (closed).
The issues, and others, discussing adding a plethora of additional attributes to the workers.
One or two extra would be fine, but as the list is extended, this will add complexity. Additionally, some of these attributes are transitional, meaning we will need to be disciplined about dismantling them after we've finished migrating code (to k8s, to object storage, etc).
Proposed Solution
Instead of adding multiple attributes, we use tags:
worker_tag_attributes :requires_shared_nfs_storage, :cron
Each call to worker_tag_attributes
is additive, meaning that it can be called multiple times, inside concerns and the main class.
Prometheus Labels
Tags do not affect prometheus labels in our monitoring stack.
Queue Selector Syntax
Queue selector syntax works as expected:
urgency=high,tags!=requires_shared_nfs_storage
Adding tags
We don't need to add any tags to current workers for this issue, but we should add tests that the queue selector works correctly in that case.
Pros and Cons
Like all tagging systems, the pros and cons are pretty much the same: flexibility being both a pro (easy to tag jobs in different ways) and a con (flexibility can be abused)
Alternatives
Alternatively, we can handle this without any further code change. We can do this by maintaining lists (for example, a list of workers that require shared NFS storage) in our configuration management system, and simply generate selectors such as worker!=BadWorker1,BadWorker2
etc.
This is fairly similar to what we do in Chef with queues today.