Define arbitrary queue attributes using tag sets

This is a fairly speculative proposal. I'm not totally sold on it, but present it as a proposal as I see several similar efforts and feel that if we were to carry them all out, implementing them all with this would be better than multiple alternatives.

The Problem

Currently we have a several proposals to add additional attributes for workers.

These include: #139 (closed) and #106 (closed).

The issues, and others, discussing adding a plethora of additional attributes to the workers.

One or two extra would be fine, but as the list is extended, this will add complexity. Additionally, some of these attributes are transitional, meaning we will need to be disciplined about dismantling them after we've finished migrating code (to k8s, to object storage, etc).

Proposed Solution

Instead of adding multiple attributes, we use tags:

worker_tag_attributes :requires_shared_nfs_storage, :cron

Each call to worker_tag_attributes is additive, meaning that it can be called multiple times, inside concerns and the main class.

Prometheus Labels

Tags do not affect prometheus labels in our monitoring stack.

Queue Selector Syntax

Queue selector syntax works as expected:

urgency=high,tags!=requires_shared_nfs_storage

Adding tags

We don't need to add any tags to current workers for this issue, but we should add tests that the queue selector works correctly in that case.

Pros and Cons

Like all tagging systems, the pros and cons are pretty much the same: flexibility being both a pro (easy to tag jobs in different ways) and a con (flexibility can be abused)

Alternatives

Alternatively, we can handle this without any further code change. We can do this by maintaining lists (for example, a list of workers that require shared NFS storage) in our configuration management system, and simply generate selectors such as worker!=BadWorker1,BadWorker2 etc.

This is fairly similar to what we do in Chef with queues today.