Skip to content

Backfill runner semver column

Pedro Pombeiro requested to merge pedropombeiro/339523/3-backfill-runners into master

What does this MR do and why?

Describe in detail what your merge request does and why.

This MR is part of a sequence of MRs that will ultimately result in being able to efficiently count the number of out-of-date CI runners in a given scope (instance/namespace/project). Please refer to the table below for more details:

Sequence MR notes
1 Add semver column to ci_runners table (!89023 - merged) Adds a migration to add indexed machine-readable semver version text column (major.minor.patch) on top of the existing ci_runners.version column.
2 Store Runner semver value on version update (!89024 - merged) Ensures that semver column is kept up-to-date based on version value.
3 Backfill runner semver column (!89054 - merged) Adds batched background migration to backfill semver column from version values.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Check that you have some ci_runner records with NULL semver values through the gdk psql -d gitlabhq_development_ci command:

    SELECT id, version, semver FROM ci_runners WHERE semver::cidr IS NULL LIMIT 10
  2. Run the migration in this branch: bundle exec rails db:migrate

  3. Check that the migration filled the semver column appropriately.

You can also check http://gdk.test:3000/admin/background_migrations?tab=finished&database=ci to see the state of the ScheduleBackfillCiRunnerSemver batched background migration (should be in the Finished tab).

Reverting the migration should remove the migration from the dashboard.

Query plans

UPDATE query

Details and visualization: https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/10569/commands/37960

EXPLAIN UPDATE
  ci_runners
SET semver = CONCAT(v.components[1]::text, '.', v.components[2]::text, '.', v.components[3]::text)
FROM (
  SELECT id AS r_id, REGEXP_SPLIT_TO_ARRAY(substring(ci_runners.version
      FROM 'v?(\d+\.\d+\.\d+)'), '\.')::smallint[] AS components
  FROM "ci_runners"
  WHERE "ci_runners"."id" BETWEEN 1 AND 2000
    AND (semver::cidr IS NULL)
    AND "ci_runners"."id" >= 1
    AND "ci_runners"."id" < 200) v
WHERE id = v.r_id
  AND v.components IS NOT NULL

Execution plan:

 ModifyTable on public.ci_runners  (cost=0.86..249.01 rows=54 width=319) (actual time=10.654..10.656 rows=0 loops=1)
   Buffers: shared hit=61 read=55 dirtied=19
   I/O Timings: read=8.070 write=0.000
   ->  Nested Loop  (cost=0.86..249.01 rows=54 width=319) (actual time=0.943..1.218 rows=1 loops=1)
         Buffers: shared hit=24 read=9 dirtied=1
         I/O Timings: read=0.543 write=0.000
         ->  Index Scan using ci_runners_pkey on public.ci_runners ci_runners_1  (cost=0.43..52.18 rows=54 width=17) (actual time=0.847..1.120 rows=1 loops=1)
               Index Cond: ((ci_runners_1.id >= 1) AND (ci_runners_1.id <= 2000) AND (ci_runners_1.id >= 1) AND (ci_runners_1.id < 200))
               Filter: (((ci_runners_1.semver)::cidr IS NULL) AND ((regexp_split_to_array("substring"((ci_runners_1.version)::text, 'v?(\d+\.\d+\.\d+)'::text), '\.'::text))::smallint[] IS NOT NULL))
               Rows Removed by Filter: 73
               Buffers: shared hit=20 read=9 dirtied=1
               I/O Timings: read=0.543 write=0.000
         ->  Index Scan using index_ci_runners_on_id_and_semver_cidr on public.ci_runners  (cost=0.43..3.45 rows=1 width=277) (actual time=0.019..0.019 rows=1 loops=1)
               Index Cond: (ci_runners.id = ci_runners_1.id)
               Buffers: shared hit=4
               I/O Timings: read=0.000 write=0.000

Background runtime estimates

Based on:

  • an estimated 1.3 million ci_runner records having no semver values
  • minimum batch size of 500 records: 2600 batches with 2 minutes interval = 3.6 days
  • maximum batch size of 10000 records: 130 batches with 2 minutes interval = 4h20m

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Production database points-of-interest

Non-conforming runner versions (will result in null version_* fields)

> SELECT DISTINCT(version) FROM ci_runners WHERE version IS NOT NULL AND semver::cidr IS NULL
+---------------------+
| version             |
|---------------------|
| 1                   |
| 1.1                 |
| 10.0                |
| HEAD                |
| HEAD-fd84d97        |
| dev                 |
| development version |
+---------------------+

Total runners not covered by migration

> SELECT COUNT(*) FROM ci_runners WHERE semver::cidr IS NULL
+-------+
| count |
|-------|
| 15250 |
+-------+

**> SELECT COUNT(*) FROM ci_runners WHERE version IS NULL
+-------+
| count |
|-------|
| 12738 |
+-------+

Part of #339523 (closed)

Edited by Pedro Pombeiro

Merge request reports