Define arbitrary queue attributes using tag sets
This is a fairly speculative proposal. I'm not totally sold on it, but present it as a proposal as I see several similar efforts and feel that if we were to carry them all out, implementing them all with this would be better than multiple alternatives.
Currently we have a several proposals to add additional attributes for workers.
These include: #139 (closed) and #106 (closed).
The issues, and others, discussing adding a plethora of additional attributes to the workers.
One or two extra would be fine, but as the list is extended, this will add complexity. Additionally, some of these attributes are transitional, meaning we will need to be disciplined about dismantling them after we've finished migrating code (to k8s, to object storage, etc).
Instead of adding multiple attributes, we use tags:
worker_tag_attributes :requires_shared_nfs_storage, :cron
Each call to
worker_tag_attributes is additive, meaning that it can be called multiple times, inside concerns and the main class.
Tags do not affect prometheus labels in our monitoring stack.
Queue Selector Syntax
Queue selector syntax works as expected:
We don't need to add any tags to current workers for this issue, but we should add tests that the queue selector works correctly in that case.
Pros and Cons
Like all tagging systems, the pros and cons are pretty much the same: flexibility being both a pro (easy to tag jobs in different ways) and a con (flexibility can be abused)
Alternatively, we can handle this without any further code change. We can do this by maintaining lists (for example, a list of workers that require shared NFS storage) in our configuration management system, and simply generate selectors such as
This is fairly similar to what we do in Chef with queues today.