GraphQL: Allow retrieving pending jobs compatible with runner
What does this MR do and why?
This MR adds a compatibleRunnerId argument to the AllJobsResolver GraphQL resolver, which allows filtering down jobs to the ones which are compatible with a specific runner configuration. It also allows filtering for pending jobs (which by definition are not yet assigned to a runner).
Changelog: added
References
Screenshots or screen recordings
How to set up and validate locally
-
In a chosen project, set up a project runner with some tags, e.g.
windows,windows-1809 -
In that project, ensure you have a job that requires the same tags.
-
Run the job.
-
Now that you have a pending job, run the GraphQL query in http://gdk.test:3000/-/graphql-explorer, adapting the runner ID in the
compatibleRunnerIdargument value:{ jobs(statuses: PENDING, compatibleRunnerId: "gid://gitlab/Ci::Runner/22978") { count nodes { id } } }
Database query plans
Pending jobs compatible with a project runner
https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/39348/commands/121141
SELECT "p_ci_builds"."status", "p_ci_builds"."finished_at", "p_ci_builds"."created_at", "p_ci_builds"."updated_at",
"p_ci_builds"."started_at", "p_ci_builds"."coverage", "p_ci_builds"."name", "p_ci_builds"."options",
"p_ci_builds"."allow_failure", "p_ci_builds"."stage_idx", "p_ci_builds"."tag", "p_ci_builds"."ref",
"p_ci_builds"."type", "p_ci_builds"."target_url", "p_ci_builds"."description", "p_ci_builds"."erased_at",
"p_ci_builds"."artifacts_expire_at", "p_ci_builds"."environment", "p_ci_builds"."when", "p_ci_builds"."yaml_variables",
"p_ci_builds"."queued_at", "p_ci_builds"."lock_version", "p_ci_builds"."coverage_regex", "p_ci_builds"."retried",
"p_ci_builds"."protected", "p_ci_builds"."failure_reason", "p_ci_builds"."scheduled_at", "p_ci_builds"."token_encrypted",
"p_ci_builds"."resource_group_id", "p_ci_builds"."waiting_for_resource_at", "p_ci_builds"."processed",
"p_ci_builds"."scheduling_type",
"p_ci_builds"."id", "p_ci_builds"."stage_id", "p_ci_builds"."partition_id", "p_ci_builds"."auto_canceled_by_partition_id",
"p_ci_builds"."auto_canceled_by_id", "p_ci_builds"."commit_id", "p_ci_builds"."erased_by_id", "p_ci_builds"."project_id",
"p_ci_builds"."runner_id", "p_ci_builds"."upstream_pipeline_id", "p_ci_builds"."user_id", "p_ci_builds"."execution_config_id",
"p_ci_builds"."upstream_pipeline_partition_id"
FROM "p_ci_builds"
JOIN (
SELECT "ci_pending_builds"."build_id", "ci_pending_builds"."partition_id"
FROM "ci_pending_builds"
WHERE "ci_pending_builds"."project_id" IN (
SELECT "ci_runner_projects"."project_id"::bigint
FROM "ci_runner_projects"
WHERE "ci_runner_projects"."runner_id" = 168793)
AND (ci_pending_builds.tag_ids <@ '{11, 541, 7166}')
AND (cardinality(tag_ids) > 0)) AS pending_subquery ON p_ci_builds.id = pending_subquery.build_id
AND p_ci_builds.partition_id = pending_subquery.partition_id
WHERE "p_ci_builds"."type" = 'Ci::Build'
AND ("p_ci_builds"."status" IN ('pending'))
ORDER BY "p_ci_builds"."id" DESC
LIMIT 101 Limit (cost=29.37..29.38 rows=1 width=1154) (actual time=132.011..132.026 rows=50 loops=1)
Buffers: shared hit=158 read=149 dirtied=52
WAL: records=87 fpi=52 bytes=420547
I/O Timings: read=124.984 write=0.000
-> Sort (cost=29.37..29.38 rows=1 width=1154) (actual time=132.009..132.020 rows=50 loops=1)
Sort Key: p_ci_builds.id DESC
Sort Method: quicksort Memory: 37kB
Buffers: shared hit=158 read=149 dirtied=52
WAL: records=87 fpi=52 bytes=420547
I/O Timings: read=124.984 write=0.000
-> Nested Loop (cost=2.98..29.36 rows=1 width=1154) (actual time=17.918..131.854 rows=50 loops=1)
Buffers: shared hit=155 read=149 dirtied=52
WAL: records=87 fpi=52 bytes=420547
I/O Timings: read=124.984 write=0.000
-> Nested Loop (cost=2.40..14.90 rows=1 width=16) (actual time=11.020..35.929 rows=50 loops=1)
Buffers: shared read=54 dirtied=37
WAL: records=71 fpi=37 bytes=304025
I/O Timings: read=31.689 write=0.000
-> Unique (cost=1.97..1.98 rows=2 width=4) (actual time=8.036..8.040 rows=1 loops=1)
Buffers: shared read=4
I/O Timings: read=7.955 write=0.000
-> Sort (cost=1.97..1.98 rows=2 width=4) (actual time=8.033..8.036 rows=1 loops=1)
Sort Key: ((ci_runner_projects.project_id)::bigint)
Sort Method: quicksort Memory: 25kB
Buffers: shared read=4
I/O Timings: read=7.955 write=0.000
-> Index Only Scan using index_unique_ci_runner_projects_on_runner_id_and_project_id on public.ci_runner_projects (cost=0.43..1.96 rows=2 width=4) (actual time=8.014..8.020 rows=1 loops=1)
Index Cond: (ci_runner_projects.runner_id = 168793)
Heap Fetches: 0
Buffers: shared read=4
I/O Timings: read=7.955 write=0.000
-> Index Scan using index_ci_pending_builds_on_project_id on public.ci_pending_builds (cost=0.42..6.45 rows=1 width=24) (actual time=2.977..27.857 rows=50 loops=1)
Index Cond: (ci_pending_builds.project_id = (ci_runner_projects.project_id)::bigint)
Filter: ((ci_pending_builds.tag_ids <@ '{11,541,7166}'::integer[]) AND (cardinality(ci_pending_builds.tag_ids) > 0))
Rows Removed by Filter: 0
Buffers: shared read=50 dirtied=37
WAL: records=71 fpi=37 bytes=304025
I/O Timings: read=23.735 write=0.000
-> Append (cost=0.58..14.41 rows=5 width=1154) (actual time=1.916..1.916 rows=1 loops=50)
Buffers: shared hit=155 read=95 dirtied=15
WAL: records=16 fpi=15 bytes=116522
I/O Timings: read=93.295 write=0.000
-> Index Scan using ci_builds_pkey on public.ci_builds p_ci_builds_1 (cost=0.58..3.61 rows=1 width=1274) (actual time=0.000..0.000 rows=0 loops=0)
Index Cond: ((p_ci_builds_1.id = ci_pending_builds.build_id) AND (p_ci_builds_1.partition_id = ci_pending_builds.partition_id))
Filter: (((p_ci_builds_1.type)::text = 'Ci::Build'::text) AND ((p_ci_builds_1.status)::text = 'pending'::text))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Index Scan using ci_builds_101_pkey on gitlab_partitions_dynamic.ci_builds_101 p_ci_builds_2 (cost=0.58..3.60 rows=1 width=600) (actual time=0.000..0.000 rows=0 loops=0)
Index Cond: ((p_ci_builds_2.id = ci_pending_builds.build_id) AND (p_ci_builds_2.partition_id = ci_pending_builds.partition_id))
Filter: (((p_ci_builds_2.type)::text = 'Ci::Build'::text) AND ((p_ci_builds_2.status)::text = 'pending'::text))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Index Scan using ci_builds_102_pkey on gitlab_partitions_dynamic.ci_builds_102 p_ci_builds_3 (cost=0.58..3.60 rows=1 width=621) (actual time=0.000..0.000 rows=0 loops=0)
Index Cond: ((p_ci_builds_3.id = ci_pending_builds.build_id) AND (p_ci_builds_3.partition_id = ci_pending_builds.partition_id))
Filter: (((p_ci_builds_3.type)::text = 'Ci::Build'::text) AND ((p_ci_builds_3.status)::text = 'pending'::text))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
-> Index Scan using ci_builds_103_pkey on gitlab_partitions_dynamic.ci_builds_103 p_ci_builds_4 (cost=0.57..3.58 rows=1 width=621) (actual time=1.913..1.913 rows=1 loops=50)
Index Cond: ((p_ci_builds_4.id = ci_pending_builds.build_id) AND (p_ci_builds_4.partition_id = ci_pending_builds.partition_id))
Filter: (((p_ci_builds_4.type)::text = 'Ci::Build'::text) AND ((p_ci_builds_4.status)::text = 'pending'::text))
Rows Removed by Filter: 0
Buffers: shared hit=155 read=95 dirtied=15
WAL: records=16 fpi=15 bytes=116522
I/O Timings: read=93.295 write=0.000
-> Seq Scan on gitlab_partitions_dynamic.ci_builds_104 p_ci_builds_5 (cost=0.00..0.00 rows=1 width=3499) (actual time=0.000..0.000 rows=0 loops=0)
Filter: (((p_ci_builds_5.type)::text = 'Ci::Build'::text) AND ((p_ci_builds_5.status)::text = 'pending'::text) AND (ci_pending_builds.build_id = p_ci_builds_5.id) AND (ci_pending_builds.partition_id = p_ci_builds_5.partition_id))
Rows Removed by Filter: 0
I/O Timings: read=0.000 write=0.000
Settings: effective_cache_size = '338688MB', seq_page_cost = '4', jit = 'off', random_page_cost = '1.5', work_mem = '100MB'MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Pedro Pombeiro