Route resumed CI jobs to correct runner machine

Ashvin Sharmarequested to merge
ashvins/route-resumed-builds-to-runner-machine into master

What does this MR do and why?

Resumed CI jobs need to land back on the exact runner machine that holds their suspended environment — otherwise the resume fails outright. Matching that at runner-poll time means comparing every candidate job against every poll, which gets slower as the queue grows. The fix is to resolve the target machine once, at queue time, and let runners filter with a single indexed predicate instead.

This MR is the first of a 6-MR stack that builds that mechanism incrementally (previously this was one large MR; splitting it lets each layer — schema, write path, read path, consumption, cleanup — be reviewed and shipped independently, which reduces the blast radius of landing a schema change and a behavior change together):

  1. This MR — schema + read-side groundwork
  2. !247560 (merged) — persist Ci::BuildRuntimeEnvironment on suspend
  3. !247561 (merged) — record the runner machine assignment when a build starts running
  4. !247562 — resolve runner_machine_id when queuing a suspend/resume build
  5. !247563 — consume the routed runner machine when assigning jobs
  6. !247564 — retire the old environment_key string-matching routing mechanism

Concretely, this MR:

  • Adds the runner_machine_id column to ci_pending_builds ahead of anything writing to it, so the schema change can roll out and bake in production on its own, decoupled from any behavior change.
  • Adds Ci::RuntimeEnvironment.find_by_key_and_project and Ci::BuildRuntimeEnvironment.runner_machine_id_for as isolated lookup helpers, unused by anything yet, so the queries themselves can be reviewed and load-tested independently of the callers that will use them in later MRs.
  • Adds a runner_machine_id predicate to the pending-builds queue query, gated behind its own derisk flag (ci_pending_builds_runner_machine_id_filter) separate from the main feature flag. Since nothing populates the column yet, this is a safe no-op today — but it lets the query shape and its performance be validated in isolation before real routing behavior goes live in a later MR.

Part of routing resumed builds to the runner machine that holds their suspended environment Resumable Jobs for CI and Agent Sessions (gitlab-org#21159).

References

Issue: Resumable Jobs for CI and Agent Sessions (gitlab-org#21159)

Blueprint: Add blueprint for suspendable environments (gitlab-com/content-sites/handbook!18954 - merged)

How to set up and validate locally

Nothing writes runner_machine_id yet in this MR, so validation is about confirming the migration and the new predicate are both safe no-ops today:

  1. Run the migration: bin/rails db:migrate
  2. Confirm the column exists: Ci::PendingBuild.column_names.include?('runner_machine_id') (in bin/rails c)
  3. Enable the derisk flag and confirm existing queue behavior is unaffected when the column is NULL for all rows: 
    Feature.enable(:ci_pending_builds_runner_machine_id_filter)
  4. Optionally, set runner_machine_id manually on a Ci::PendingBuild row in the console and confirm Ci::Queue::PendingBuildsStrategy#builds_for_runner_manager excludes it for a runner_manager with a different id, and includes it for a matching one or nil.

For database reviewers

Query plan for using the runner_machine_id predicate to locate pending builds for a runner machine:

Click here to see 
[1] pry(main)> plan = Ci::RuntimeEnvironment
.where(project_id: 42, environment_key: "does-not-exist")
.limit(1)
  Ci::RuntimeEnvironment Load (3.8ms)  SELECT "p_ci_runtime_environments"."id", "p_ci_runtime_environments"."project_id", "p_ci_runtime_environments"."created_at", "p_ci_runtime_environments"."updated_at", "p_ci_runtime_environments"."environment_key" FROM "p_ci_runtime_environments" WHERE "p_ci_runtime_environments"."project_id" = 42 AND "p_ci_runtime_environments"."environment_key" = 'does-not-exist' LIMIT 1 /*application:console,db_config_database:gitlabhq_development_ci,db_config_name:ci,console_hostname:ashsharma--20241227-M9QNN,console_username:ashvin,line:<internal:kernel>:187:in `loop'*/
=> EXPLAIN (ANALYZE, BUFFERS) SELECT "p_ci_runtime_environments"."id", "p_ci_runtime_environments"."project_id", "p_ci_runtime_environments"."created_at", "p_ci_runtime_environments"."updated_at", "p_ci_runtime_environments"."environment_key" FROM "p_ci_runtime_environments" WHERE "p_ci_runtime_environments"."project_id" = 42 AND "p_ci_runtime_environments"."environment_key" = 'does-not-exist' LIMIT 1 /*application:console,db_config_database:gitlabhq_development_ci,db_config_name:ci,console_hostname:ashsharma--20241227-M9QNN,console_username:ashvin,line:<internal:kernel>:187:in `loop'*/
                                                                                       QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.54..2.50 rows=1 width=244) (actual time=0.012..0.013 rows=0 loops=1)
   Buffers: shared hit=30
   ->  Append  (cost=0.54..20.16 rows=10 width=244) (actual time=0.012..0.013 rows=0 loops=1)
         Buffers: shared hit=30
         ->  Index Scan using index_5d21a827a2 on p_ci_runtime_environments_29 p_ci_runtime_environments_1  (cost=0.54..2.56 rows=1 width=310) (actual time=0.002..0.002 rows=0 loops=1)
               Index Cond: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Buffers: shared hit=4
         ->  Index Scan using index_9d154dc17f on p_ci_runtime_environments_30 p_ci_runtime_environments_2  (cost=0.54..2.56 rows=1 width=310) (actual time=0.001..0.001 rows=0 loops=1)
               Index Cond: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Buffers: shared hit=4
         ->  Index Scan using index_26ab6a7746 on p_ci_runtime_environments_31 p_ci_runtime_environments_3  (cost=0.54..2.56 rows=1 width=312) (actual time=0.001..0.001 rows=0 loops=1)
               Index Cond: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Buffers: shared hit=4
         ->  Index Scan using index_ef182c59c0 on p_ci_runtime_environments_32 p_ci_runtime_environments_4  (cost=0.54..2.56 rows=1 width=309) (actual time=0.001..0.001 rows=0 loops=1)
               Index Cond: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Buffers: shared hit=4
         ->  Index Scan using index_2fc9597b3b on p_ci_runtime_environments_33 p_ci_runtime_environments_5  (cost=0.54..2.56 rows=1 width=310) (actual time=0.001..0.001 rows=0 loops=1)
               Index Cond: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Buffers: shared hit=4
         ->  Index Scan using index_3688fcaad3 on p_ci_runtime_environments_34 p_ci_runtime_environments_6  (cost=0.54..2.56 rows=1 width=311) (actual time=0.001..0.001 rows=0 loops=1)
               Index Cond: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Buffers: shared hit=4
         ->  Index Scan using index_4bd04548b6 on p_ci_runtime_environments_35 p_ci_runtime_environments_7  (cost=0.54..2.56 rows=1 width=309) (actual time=0.001..0.001 rows=0 loops=1)
               Index Cond: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Buffers: shared hit=4
         ->  Seq Scan on p_ci_runtime_environments_36 p_ci_runtime_environments_8  (cost=0.00..1.04 rows=1 width=105) (actual time=0.002..0.002 rows=0 loops=1)
               Filter: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Rows Removed by Filter: 3
               Buffers: shared hit=1
         ->  Seq Scan on p_ci_runtime_environments_37 p_ci_runtime_environments_9  (cost=0.00..1.17 rows=1 width=105) (actual time=0.001..0.001 rows=0 loops=1)
               Filter: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
               Rows Removed by Filter: 11
               Buffers: shared hit=1
         ->  Seq Scan on p_ci_runtime_environments_38 p_ci_runtime_environments_10  (cost=0.00..0.00 rows=1 width=64) (actual time=0.000..0.000 rows=0 loops=1)
               Filter: ((project_id = 42) AND (environment_key = 'does-not-exist'::text))
 Planning:
   Buffers: shared hit=3
 Planning Time: 0.109 ms
 Execution Time: 0.025 ms
(39 rows)

[2] pry(main)> Ci::RuntimeEnvironment.where(project_id: 42).count
  Ci::RuntimeEnvironment Count (17.7ms)  SELECT COUNT(*) FROM "p_ci_runtime_environments" WHERE "p_ci_runtime_environments"."project_id" = 42 /*application:console,db_config_database:gitlabhq_development_ci,db_config_name:ci,console_hostname:ashsharma--20241227-M9QNN,console_username:ashvin,line:(pry):5:in `__pry__'*/
=> 37826```

</details>

## MR acceptance checklist

Evaluate this MR against the [MR acceptance checklist](https://docs.gitlab.com/development/code_review/#acceptance-checklist). It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Ashvin Sharma

Merge request reports