Skip to content

GitLab Next

  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • scalability scalability
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 245
    • Issues 245
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar

GitLab 15.0 is launching on May 22! This version brings many exciting improvements, but also removes deprecated features and introduces breaking changes that may impact your workflow. To see what is being deprecated and removed, please visit Breaking changes in 15.0 and Deprecations.

  • GitLab.com
  • GitLab Infrastructure Team
  • scalabilityscalability
  • Issues
  • #261
Closed
Open
Created Mar 31, 2020 by Andrew Newdigate@andrewnOwner

Define arbitrary queue attributes using tag sets

This is a fairly speculative proposal. I'm not totally sold on it, but present it as a proposal as I see several similar efforts and feel that if we were to carry them all out, implementing them all with this would be better than multiple alternatives.

The Problem

Currently we have a several proposals to add additional attributes for workers.

These include: #139 (closed) and #106 (closed).

The issues, and others, discussing adding a plethora of additional attributes to the workers.

One or two extra would be fine, but as the list is extended, this will add complexity. Additionally, some of these attributes are transitional, meaning we will need to be disciplined about dismantling them after we've finished migrating code (to k8s, to object storage, etc).

Proposed Solution

Instead of adding multiple attributes, we use tags:

worker_tag_attributes :requires_shared_nfs_storage, :cron

Each call to worker_tag_attributes is additive, meaning that it can be called multiple times, inside concerns and the main class.

Prometheus Labels

Tags do not affect prometheus labels in our monitoring stack.

Queue Selector Syntax

Queue selector syntax works as expected:

urgency=high,tags!=requires_shared_nfs_storage

Adding tags

We don't need to add any tags to current workers for this issue, but we should add tests that the queue selector works correctly in that case.

Pros and Cons

Like all tagging systems, the pros and cons are pretty much the same: flexibility being both a pro (easy to tag jobs in different ways) and a con (flexibility can be abused)

Alternatives

Alternatively, we can handle this without any further code change. We can do this by maintaining lists (for example, a list of workers that require shared NFS storage) in our configuration management system, and simply generate selectors such as worker!=BadWorker1,BadWorker2 etc.

This is fairly similar to what we do in Chef with queues today.

Edited May 06, 2020 by Sean McGivern
Assignee
Assign to
Time tracking