Skip to content

Backfill project import level on namespace settings

Carla Drago requested to merge 358750-backfill-project-import-level into master

What does this MR do and why?

This batched background migration aligns the project_import_level in existing Group namespaces to their corresponding values for project_creation_level.

This is to avoid a breaking change when the ability to configure project import levels go live.

See #358750 (closed)

See this issue comment around why we're making this change.

Sub-Batch Update Query

explain UPDATE "namespace_settings" SET "project_import_level" = 30 WHERE "namespace_settings"."namespace_id" IN (SELECT "namespaces"."id" FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND "namespaces"."project_creation_level" IN (2, 3, 4, 5) LIMIT 100)

Plan with Execution

ModifyTable on public.namespace_settings  (cost=20160.34..20506.16 rows=100 width=130) (actual time=777.221..777.225 rows=0 loops=1)
   Buffers: shared hit=4399 read=1391 dirtied=182
   I/O Timings: read=734.220 write=0.000
   ->  Nested Loop  (cost=20160.34..20506.16 rows=100 width=130) (actual time=681.481..730.595 rows=100 loops=1)
         Buffers: shared hit=3695 read=1296 dirt

Summary

Time: 777.883 ms
  - planning: 0.550 ms
  - execution: 777.333 ms
    - I/O read: 734.220 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 1049759 (~8.00 GiB) from the buffer pool
  - reads: 126086 (~985.00 MiB) from the OS file cache, including disk I/O
  - dirtied: 20201 (~157.80 MiB)
  - writes: 0

Rows Affected

explain SELECT id FROM "namespaces" WHERE "namespaces"."type"='Group' AND "namespaces"."project_creation_level" IN (0, 2, 3, 4, 5);
 Gather  (cost=1000.43..1241036.56 rows=6838 width=4) 
         (actual time=68.188..201608.887 **rows=74718** loops=1)
   Workers Planned: 2
   Workers Launched: 2
   Buffers: shared hit=2235347 read=1074470 dirtied=16366
   I/O Timings: read=584775.754 write=0.000
   ->  Parallel Index Scan using index_groups_on_parent_id_id on public.namespaces  (cost=0.43..1239352.76 rows=2849 width=4) (actual

~75K (74718)

Estimated Execution Time

~ 39 minutes

Breakdown:

  • per sub-batch (4 queries): 777.883ms * 4 = 3111.532ms

  • per batch (10 sub-batches per batch): 3111.532ms * 10 = 31115.32ms

  • total (75 batches required to process 75K records): 31115.32ms * 75 = 2333649ms

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #358750 (closed)

Edited by Carla Drago

Merge request reports