Skip to content

Backfill failed imported snippet repositories

Francisco Javier López (ex-Gitlab) requested to merge fj-backfill-imported-snippet-repositories into master

What does this MR do?

Refs #220767 (closed)

After fixing all the problems with failed project imports in #219752 (closed) that affected the creation of snippet repositories, we need to update those remaining ones that we have left.

In production, we have 130 records.

This operation will be done in a background migration. To avoid the record distribution problem and enqueue the background migration from id 10 to 500, for example, since the number of records is small, we split them in consecutive batches and run the background migration on each batch.

In the worst-case scenario where each batch contains only one snippet, the time would be 130*2 = 260s => 4.33 hours

Query plan

explain SELECT "snippets".* FROM "snippets" WHERE "snippets"."id" NOT IN (SELECT "snippet_repositories"."snippet_id" FROM "snippet_repositories")
Seq Scan on public.snippets  (cost=7768.65..182683.25 rows=137704 width=2089) (actual time=169.704..339.007 rows=130 loops=1)
   Filter: (NOT (hashed SubPlan 1))
   Rows Removed by Filter: 275367
   Buffers: shared hit=64398
   SubPlan 1
     ->  Index Only Scan using snippet_repositories_pkey on public.snippet_repositories  (cost=0.42..7081.32 rows=274931 width=8) (actual time=0.028..51.541 rows=275367 loops=1)
           Heap Fetches: 25512
           Buffers: shared hit=21530
Time: 340.880 ms
  - planning: 0.161 ms
  - execution: 340.719 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 64398 (~503.10 MiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

Migrations running

Up

bundle exec rake db:migrate
== 20200608072931 BackfillImportedSnippetRepositories: migrating ==============
== 20200608072931 BackfillImportedSnippetRepositories: migrated (0.0214s) =====

Down

bundle exec rake db:rollback
== 20200608072931 BackfillImportedSnippetRepositories: reverting ==============
== 20200608072931 BackfillImportedSnippetRepositories: reverted (0.0000s) =====

Does this MR meet the acceptance criteria?

Conformity

Edited by 🤖 GitLab Bot 🤖

Merge request reports