Enable load balancing for BuildQueueWorker [RUN ALL RSPEC] [RUN AS-IF-FOSS]

What does this MR do?

Implements #330681 (closed).
Set behind the feature flag: load_balancing_for_build_queue_worker and it is disabled by default.
Rollout issue: #332586 (closed)

Does this MR meet the acceptance criteria?

Conformity

• I have included a changelog entry, or it's not needed. (Does this MR need a changelog?)
• I have added/updated documentation, or it's not needed. (Is documentation required?)
• I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
• I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
• I have self-reviewed this MR per code review guidelines.
• This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
• I have followed the style guides.

Availability and Testing

Injecting fake SELECTs into beginning and the end of perfrom:

dev logs (collapsed)

   (1.5ms)  SELECT 'TANUKI_QUEUE_START' /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/workers/build_queue_worker.rb:18:in `perform'*/
  ↳ app/workers/build_queue_worker.rb:18:in `perform'
  Group Load (4.7ms)  SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" = 25 LIMIT 1 /*application:sidekiq,correlation_id:48b3ce68450eae264eae26b301495a1a,jid:c0a305f52333dea2fd390042,endpoint_id:BuildHooksWorker,line:/ee/app/models/ee/project.rb:437:in `has_group_hooks?'*/
  ↳ ee/app/models/ee/project.rb:437:in `has_group_hooks?'
  Ci::Build Load (5.1ms)  SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."id" = 967 LIMIT 1 /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/workers/build_queue_worker.rb:19:in `perform'*/
  ↳ app/workers/build_queue_worker.rb:19:in `perform'
  License Load (5.9ms)  SELECT "licenses".* FROM "licenses" ORDER BY "licenses"."id" DESC LIMIT 100 /*application:sidekiq,correlation_id:48b3ce68450eae264eae26b301495a1a,jid:c0a305f52333dea2fd390042,endpoint_id:BuildHooksWorker,line:/ee/app/models/license.rb:302:in `load_license'*/
  ↳ ee/app/models/license.rb:302:in `load_license'
  Project Load (4.0ms)  SELECT "projects".* FROM "projects" WHERE "projects"."id" = 5 LIMIT 1 /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/services/ci/update_build_queue_service.rb:55:in `tick'*/
  ↳ app/services/ci/update_build_queue_service.rb:55:in `tick'
  Integration Exists? (1.9ms)  SELECT 1 AS one FROM "services" WHERE "services"."project_id" = 5 AND "services"."job_events" = TRUE AND "services"."active" = TRUE LIMIT 1 /*application:sidekiq,correlation_id:48b3ce68450eae264eae26b301495a1a,jid:c0a305f52333dea2fd390042,endpoint_id:BuildHooksWorker,line:/app/models/project.rb:1512:in `has_active_services?'*/
  ↳ app/models/project.rb:1512:in `has_active_services?'
  ProjectCiCdSetting Load (9.7ms)  SELECT "project_ci_cd_settings".* FROM "project_ci_cd_settings" WHERE "project_ci_cd_settings"."project_id" = 5 LIMIT 1 /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/models/project.rb:426:in `group_runners_enabled?'*/
  ↳ app/models/project.rb:426:in `group_runners_enabled?'
  Namespace Load (3.0ms)  SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."id" = 25 LIMIT 1 /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/models/project.rb:2545:in `root_namespace'*/
  ↳ app/models/project.rb:2545:in `root_namespace'
  Ci::Pipeline Load (2.4ms)  SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 13 LIMIT 1 /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/app/workers/pipeline_process_worker.rb:19:in `perform'*/
  ↳ app/workers/pipeline_process_worker.rb:19:in `perform'
  Project Exists? (4.1ms)  SELECT 1 AS one FROM "projects" WHERE "projects"."namespace_id" IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" = 25)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" = 'Group' AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "id" FROM "base_and_descendants" AS "namespaces") AND "projects"."shared_runners_enabled" = TRUE LIMIT 1 /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/ee/app/models/ee/namespace.rb:250:in `any_project_with_shared_runners_enabled?'*/
  ↳ ee/app/models/ee/namespace.rb:250:in `any_project_with_shared_runners_enabled?'
  Project Load (2.5ms)  SELECT "projects".* FROM "projects" WHERE "projects"."id" = 5 LIMIT 1 /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/app/services/ci/process_pipeline_service.rb:33:in `update_retried'*/
  ↳ app/services/ci/process_pipeline_service.rb:33:in `update_retried'
  ApplicationSetting Load (3.4ms)  SELECT "application_settings".* FROM "application_settings" ORDER BY "application_settings"."id" DESC LIMIT 1 /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/models/concerns/cacheable_attributes.rb:19:in `current_without_cache'*/
  ↳ app/models/concerns/cacheable_attributes.rb:19:in `current_without_cache'
   (4.5ms)  SELECT MAX(id), "ci_builds"."name" FROM "ci_builds" WHERE "ci_builds"."commit_id" = 13 AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) GROUP BY "ci_builds"."name" HAVING (count(*) > 1) /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/app/services/ci/process_pipeline_service.rb:39:in `update_retried'*/
  ↳ app/services/ci/process_pipeline_service.rb:39:in `update_retried'
  CommitStatus Exists? (0.9ms)  SELECT 1 AS one FROM "ci_builds" WHERE "ci_builds"."commit_id" = 13 AND ("ci_builds"."processed" = FALSE OR "ci_builds"."processed" IS NULL) AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) LIMIT 1 /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/app/models/ci/pipeline.rb:702:in `needs_processing?'*/
  ↳ app/models/ci/pipeline.rb:702:in `needs_processing?'
  Ci::Stage Load (2.1ms)  SELECT "ci_stages".* FROM "ci_stages" WHERE "ci_stages"."pipeline_id" = 13 ORDER BY "ci_stages"."position" ASC /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/app/services/ci/pipeline_processing/atomic_processing_service.rb:43:in `update_stages!'*/
  ↳ app/services/ci/pipeline_processing/atomic_processing_service.rb:43:in `update_stages!'
  Ci::Runner Load (6.6ms)  SELECT "ci_runners".* FROM ((SELECT "ci_runners".* FROM "ci_runners" INNER JOIN "ci_runner_projects" ON "ci_runners"."id" = "ci_runner_projects"."runner_id" WHERE "ci_runner_projects"."project_id" = 5)
UNION
(SELECT "ci_runners".* FROM "ci_runners" INNER JOIN "ci_runner_namespaces" ON "ci_runner_namespaces"."runner_id" = "ci_runners"."id" INNER JOIN "namespaces" ON "namespaces"."id" = "ci_runner_namespaces"."namespace_id" AND "namespaces"."type" = 'Group' WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces".* FROM "namespaces" INNER JOIN "projects" ON "projects"."namespace_id" = "namespaces"."id" WHERE "namespaces"."type" = 'Group' AND "projects"."id" = 5)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_ancestors" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" = "base_and_ancestors"."parent_id")) SELECT "namespaces"."id" FROM "base_and_ancestors" AS "namespaces"))
UNION
(SELECT "ci_runners".* FROM "ci_runners" WHERE "ci_runners"."runner_type" = 1)) ci_runners WHERE (contacted_at > '2021-06-03 11:48:26.474249') /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/services/ci/update_build_queue_service.rb:66:in `tick_for'*/
  ↳ app/services/ci/update_build_queue_service.rb:66:in `tick_for'
   (2.0ms)  SELECT "ci_builds"."id", "ci_builds"."name", "ci_builds"."status", "ci_builds"."allow_failure", "ci_builds"."stage_idx", "ci_builds"."processed", "ci_builds"."lock_version" FROM "ci_builds" WHERE "ci_builds"."commit_id" = 13 AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) ORDER BY "ci_builds"."stage_idx" ASC /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/app/services/ci/pipeline_processing/atomic_processing_service/status_collection.rb:124:in `block in all_statuses'*/
  ↳ app/services/ci/pipeline_processing/atomic_processing_service/status_collection.rb:124:in `block in all_statuses'
  ActsAsTaggableOn::Tagging Load (3.9ms)  SELECT "taggings".* FROM "taggings" WHERE "taggings"."taggable_type" = 'Ci::Runner' AND "taggings"."context" = 'tags' AND "taggings"."taggable_id" = 1 /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/services/ci/update_build_queue_service.rb:66:in `tick_for'*/
  ↳ app/services/ci/update_build_queue_service.rb:66:in `tick_for'
  CACHE Project Load (0.0ms)  SELECT "projects".* FROM "projects" WHERE "projects"."id" = 5 LIMIT 1
  ↳ lib/gitlab/database.rb:376:in `block in transaction'
  TRANSACTION (5.2ms)  BEGIN /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/lib/gitlab/database.rb:376:in `block in transaction'*/
  ↳ lib/gitlab/database.rb:376:in `block in transaction'
  ActsAsTaggableOn::Tagging Load (1.3ms)  SELECT "taggings".* FROM "taggings" WHERE "taggings"."taggable_id" = 967 AND "taggings"."taggable_type" = 'CommitStatus' /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/models/ci/build.rb:404:in `block in build_matcher'*/
  ↳ app/models/ci/build.rb:404:in `block in build_matcher'
  Ci::Stage Update (1.9ms)  UPDATE "ci_stages" SET "updated_at" = '2021-06-03 14:48:26.602314', "status" = 2, "lock_version" = 6 WHERE "ci_stages"."id" = 50 AND "ci_stages"."lock_version" = 5 /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/lib/gitlab/database.rb:376:in `block in transaction'*/
  ↳ lib/gitlab/database.rb:376:in `block in transaction'
  TRANSACTION (0.9ms)  COMMIT /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/lib/gitlab/database.rb:376:in `block in transaction'*/
  ↳ lib/gitlab/database.rb:376:in `block in transaction'
  ActsAsTaggableOn::Tag Load (0.8ms)  SELECT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."taggable_id" = 967 AND "taggings"."taggable_type" = 'CommitStatus' AND (taggings.context = 'tags' AND taggings.tagger_id IS NULL) /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/models/ci/build.rb:404:in `block in build_matcher'*/
  ↳ app/models/ci/build.rb:404:in `block in build_matcher'
  TRANSACTION (3.2ms)  BEGIN /*application:sidekiq,correlation_id:87a8750aa29d60d604fcfcea678967d2,jid:70823ff4f2b8ac01beac718a,endpoint_id:PipelineProcessWorker,line:/lib/gitlab/database.rb:376:in `block in transaction'*/
  ↳ lib/gitlab/database.rb:376:in `block in transaction'
   (0.1ms)  SELECT 'TANUKI_QUEUE_END' /*application:sidekiq,correlation_id:7c43418997c6b7266a435630522d4808,jid:5741102189d3b9e790f5eb7e,endpoint_id:BuildQueueWorker,line:/app/workers/build_queue_worker.rb:22:in `perform'*/

• I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
• I have tested this MR in all supported browsers, or it's not needed.
• I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

N/A

Related to #330681 (closed)

app/workers/build_queue_worker.rb

@@ -10,6 +10,7 @@ class BuildQueueWorker # rubocop:disable Scalability/IdempotentWorker
   feature_category :continuous_integration
   urgency :high
   worker_resource_boundary :cpu
+  data_consistency :sticky, feature_flag: :load_balancing_for_build_queue_worker
 
   # rubocop: disable CodeReuse/ActiveRecord
   def perform(build_id)
     Ci::Build.find_by(id: build_id).try do |build|
       Ci::UpdateBuildQueueService.new.tick(build)
     end
   end
   # rubocop: enable CodeReuse/ActiveRecord