Migrate all merge request user mentions to db table (!34378) · Merge requests · GitLab.org / GitLab

Alexandru Croitor requested to merge migrate-all-merge-request-user-mentions into master Jun 11, 2020

What does this MR do?

Approach 1 - Sometimes fails with timeout

We did see query timeouts in some of the previous mentions migrations, when we tried to fetch e.g. 1000 records that contain @ character, because records with mentions are not uniformly distributed throughout the entire table, thus resulting sometimes in having to scan millions or tens of millions of records in order to fetch those 1000 records that contain @ character.

Approach 2 - Current attempt, that should not timeout, but may take longer

This migration approach instead scans the entire merge_requests table in batches of 100K and then select only records that can contain mentions from each of the 100K batch. This way the number of fetched records is constantly 100K and we can ensure query does not timeout due to an overly large batch of records being queried.

Downside is we will have more migration background jobs than with Approach 1, but this should prevent the migration from timing out, thus failing.

Queries:

Merge_request table size on Gitlab.com from #database-lab: ~~50042634~~ updated: 53324164

EXPLAIN SELECT id FROM merge_requests

Index Only Scan using merge_requests_pkey on public.merge_requests  (cost=0.56..1577827.95 rows=50057580 width=4) (actual time=9.847..337771.599 rows=50042634 loops=1)
   Heap Fetches: 1551062
   Buffers: shared hit=25312510 read=520420 dirtied=70064 written=14
   I/O Timings: read=311631.023 write=1.042

~~Merge Requests 50_042_634 / 100_000 => ~500 migration background jobs~~
updated: 53_324_164 / 100_000 => ~533

Migration SQLs & Plans

Fetch batches of 100_000 records from `merge_requests`, several different queries with different offset by ID.

Click to see SQLs and plans


  EXPLAIN SELECT "merge_requests"."id" FROM "merge_requests" 
  WHERE "merge_requests"."id" >= 1 ORDER BY "merge_requests"."id" ASC LIMIT 1 OFFSET 100000


  Limit  (cost=3402.59..3402.62 rows=1 width=4) (actual time=98.645..98.646 rows=1 loops=1)
     Buffers: shared hit=45646
     ->  Index Only Scan using merge_requests_pkey on public.merge_requests  (cost=0.56..1702971.90 rows=50057580 width=4) (actual time=0.025..91.530 rows=100001 loops=1)
           Index Cond: (merge_requests.id >= 1)
           Heap Fetches: 3292
           Buffers: shared hit=45646
  
  Time: 98.968 ms
    - planning: 0.265 ms
    - execution: 98.703 ms
      - I/O read: 0.000 ms
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 45646 (~356.60 MiB) from the buffer pool
    - reads: 0 from the OS file cache, including disk I/O
    - dirtied: 0
    - writes: 0


  EXPLAIN SELECT "merge_requests"."id" FROM "merge_requests" 
  WHERE "merge_requests"."id" >= 10900000 ORDER BY "merge_requests"."id" ASC LIMIT 1 OFFSET 100000


  Limit  (cost=3420.54..3420.57 rows=1 width=4) (actual time=845.994..845.995 rows=1 loops=1)
     Buffers: shared hit=36324 read=827
     I/O Timings: read=701.641
     ->  Index Only Scan using merge_requests_pkey on public.merge_requests  (cost=0.56..1429689.09 rows=41804099 width=4) (actual time=6.376..837.493 rows=100001 loops=1)
           Index Cond: (merge_requests.id >= 10900000)
           Heap Fetches: 2530
           Buffers: shared hit=36324 read=827
           I/O Timings: read=701.641
  
  Time: 846.341 ms
    - planning: 0.312 ms
    - execution: 846.029 ms
      - I/O read: 701.641 ms
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 36324 (~283.80 MiB) from the buffer pool
    - reads: 827 (~6.50 MiB) from the OS file cache, including disk I/O
    - dirtied: 0
    - writes: 0


  EXPLAIN SELECT "merge_requests"."id" FROM "merge_requests" 
  WHERE "merge_requests"."id" >= 49900000 ORDER BY "merge_requests"."id" ASC LIMIT 1 OFFSET 100000


  Limit  (cost=3743.15..3743.19 rows=1 width=4) (actual time=108.853..108.854 rows=1 loops=1)
     Buffers: shared hit=46261 read=2
     I/O Timings: read=12.687
     ->  Index Only Scan using merge_requests_pkey on public.merge_requests  (cost=0.56..394642.75 rows=10544629 width=4) (actual time=12.899..100.712 rows=100001 loops=1)
           Index Cond: (merge_requests.id >= 49900000)
           Heap Fetches: 828
           Buffers: shared hit=46261 read=2
           I/O Timings: read=12.687
  
  Time: 109.218 ms
    - planning: 0.319 ms
    - execution: 108.899 ms
      - I/O read: 12.687 ms
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 46261 (~361.40 MiB) from the buffer pool
    - reads: 2 (~16.00 KiB) from the OS file cache, including disk I/O
    - dirtied: 0
    - writes: 0

Check that fetched batch contains at least one record with `@` character and was not already saved in `merge_request_user_mentions` table. This should help reduce the number of necessary background migration jobs.

Click to see the plans


  EXPLAIN SELECT COUNT(*) FROM "merge_requests" 
  LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id 
  WHERE ((description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL) 
  AND "merge_requests"."id" BETWEEN 1 AND 100000


  Aggregate  (cost=2037.28..2037.29 rows=1 width=8) (actual time=37.076..37.076 rows=1 loops=1)
     Buffers: shared hit=9341 read=30
     I/O Timings: read=28.532
     ->  Nested Loop Anti Join  (cost=0.86..2032.41 rows=1948 width=0) (actual time=8.231..36.965 rows=541 loops=1)
           Buffers: shared hit=9341 read=30
           I/O Timings: read=28.532
           ->  Index Only Scan using merge_request_mentions_temp_index on public.merge_requests  (cost=0.43..97.41 rows=1976 width=4) (actual time=8.145..22.446 rows=2594 loops=1)
                 Index Cond: ((merge_requests.id >= 1) AND (merge_requests.id <= 100000))
                 Heap Fetches: 83
                 Buffers: shared hit=1555 read=16
                 I/O Timings: read=20.410
           ->  Index Only Scan using merge_request_user_mentions_on_mr_id_and_note_id_index on public.merge_request_user_mentions  (cost=0.43..2.07 rows=3 width=4) (actual time=0.005..0.005 rows=1 loops=2594)
                 Index Cond: (merge_request_user_mentions.merge_request_id = merge_requests.id)
                 Heap Fetches: 0
                 Buffers: shared hit=7786 read=14
                 I/O Timings: read=8.122
  
  Time: 42.193 ms
    - planning: 5.044 ms
    - execution: 37.149 ms
      - I/O read: 28.532 ms
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 9341 (~73.00 MiB) from the buffer pool
    - reads: 30 (~240.00 KiB) from the OS file cache, including disk I/O
    - dirtied: 0
    - writes: 0


  EXPLAIN SELECT COUNT(*) FROM "merge_requests" 
  LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id 
  WHERE ((description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL) 
  AND "merge_requests"."id" BETWEEN 19900000 AND 20000000


  Aggregate  (cost=2021.47..2021.48 rows=1 width=8) (actual time=25.710..25.710 rows=1 loops=1)
     Buffers: shared hit=4591 read=11
     I/O Timings: read=13.440
     ->  Nested Loop Anti Join  (cost=0.86..2016.64 rows=1932 width=0) (actual time=7.265..25.564 rows=639 loops=1)
           Buffers: shared hit=4591 read=11
           I/O Timings: read=13.440
           ->  Index Only Scan using merge_request_mentions_temp_index on public.merge_requests  (cost=0.43..96.40 rows=1960 width=4) (actual time=2.866..14.071 rows=1296 loops=1)
                 Index Cond: ((merge_requests.id >= 19900000) AND (merge_requests.id <= 20000000))
                 Heap Fetches: 43
                 Buffers: shared hit=701 read=5
                 I/O Timings: read=6.018
           ->  Index Only Scan using merge_request_user_mentions_on_mr_id_and_note_id_index on public.merge_request_user_mentions  (cost=0.43..2.07 rows=3 width=4) (actual time=0.008..0.008 rows=1 loops=1296)
                 Index Cond: (merge_request_user_mentions.merge_request_id = merge_requests.id)
                 Heap Fetches: 0
                 Buffers: shared hit=3890 read=6
                 I/O Timings: read=7.422
  
  Time: 31.289 ms
    - planning: 5.516 ms
    - execution: 25.773 ms
      - I/O read: 13.440 ms
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 4591 (~35.90 MiB) from the buffer pool
    - reads: 11 (~88.00 KiB) from the OS file cache, including disk I/O
    - dirtied: 0
    - writes: 0


  EXPLAIN SELECT COUNT(*) FROM "merge_requests" 
  LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id 
  WHERE ((description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL) 
  AND "merge_requests"."id" BETWEEN 48600000 AND 48700000


  Aggregate  (cost=2279.10..2279.11 rows=1 width=8) (actual time=86.825..86.826 rows=1 loops=1)
     Buffers: shared hit=6999 read=52 dirtied=3
     I/O Timings: read=74.100
     ->  Nested Loop Anti Join  (cost=0.86..2273.62 rows=2192 width=0) (actual time=7.448..86.676 rows=752 loops=1)
           Buffers: shared hit=6999 read=52 dirtied=3
           I/O Timings: read=74.100
           ->  Index Only Scan using merge_request_mentions_temp_index on public.merge_requests  (cost=0.43..108.84 rows=2224 width=4) (actual time=4.608..12.764 rows=1929 loops=1)
                 Index Cond: ((merge_requests.id >= 48600000) AND (merge_requests.id <= 48700000))
                 Heap Fetches: 10
                 Buffers: shared hit=1237 read=9
                 I/O Timings: read=9.341
           ->  Index Only Scan using merge_request_user_mentions_on_mr_id_and_note_id_index on public.merge_request_user_mentions  (cost=0.43..2.05 rows=3 width=4) (actual time=0.038..0.038 rows=1 loops=1929)
                 Index Cond: (merge_request_user_mentions.merge_request_id = merge_requests.id)
                 Heap Fetches: 5
                 Buffers: shared hit=5762 read=43 dirtied=3
                 I/O Timings: read=64.759
  
  Time: 91.281 ms
    - planning: 4.398 ms
    - execution: 86.883 ms
      - I/O read: 74.100 ms
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 6999 (~54.70 MiB) from the buffer pool
    - reads: 52 (~416.00 KiB) from the OS file cache, including disk I/O
    - dirtied: 3 (~24.00 KiB)
    - writes: 0

Fetch records that contain `@` character and have not yet been migrated from 100K batch

Click to see the plans


  EXPLAIN SELECT "merge_requests".* FROM "merge_requests" 
  LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id 
  WHERE ((description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL) 
  AND "merge_requests"."id" BETWEEN 1 AND 100000


  Nested Loop Anti Join  (cost=0.86..3628.32 rows=1948 width=732) (actual time=14.542..3943.857 rows=541 loops=1)
     Buffers: shared hit=8207 read=1942 dirtied=1
     I/O Timings: read=3801.519
     ->  Index Scan using merge_request_mentions_temp_index on public.merge_requests  (cost=0.43..1693.32 rows=1976 width=732) (actual time=14.476..3895.043 rows=2594 loops=1)
           Index Cond: ((merge_requests.id >= 1) AND (merge_requests.id <= 100000))
           Buffers: shared hit=407 read=1942 dirtied=1
           I/O Timings: read=3801.519
     ->  Index Only Scan using merge_request_user_mentions_on_mr_id_and_note_id_index on public.merge_request_user_mentions  (cost=0.43..2.07 rows=3 width=4) (actual time=0.013..0.013 rows=1 loops=2594)
           Index Cond: (merge_request_user_mentions.merge_request_id = merge_requests.id)
           Heap Fetches: 0
           Buffers: shared hit=7800

  Time: 3.949 s
    - planning: 4.812 ms
    - execution: 3.944 s
      - I/O read: 3.802 s
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 8207 (~64.10 MiB) from the buffer pool
    - reads: 1942 (~15.20 MiB) from the OS file cache, including disk I/O
    - dirtied: 1 (~8.00 KiB)
    - writes: 0


  EXPLAIN SELECT "merge_requests".* FROM "merge_requests"
  LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id
  WHERE ((description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL)
  AND "merge_requests"."id" BETWEEN 15300000 AND 15400000


  Nested Loop Anti Join  (cost=0.86..3642.18 rows=1954 width=732) (actual time=25.601..3020.348 rows=812 loops=1)
     Buffers: shared hit=4345 read=1360 dirtied=5
     I/O Timings: read=2936.930
     ->  Index Scan using merge_request_mentions_temp_index on public.merge_requests  (cost=0.43..1700.47 rows=1983 width=732) (actual time=9.782..2965.625 rows=1431 loops=1)
           Index Cond: ((merge_requests.id >= 15300000) AND (merge_requests.id <= 15400000))
           Buffers: shared hit=49 read=1354 dirtied=5
           I/O Timings: read=2916.616
     ->  Index Only Scan using merge_request_user_mentions_on_mr_id_and_note_id_index on public.merge_request_user_mentions  (cost=0.43..2.07 rows=3 width=4) (actual time=0.030..0.030 rows=0 loops=1431)
           Index Cond: (merge_request_user_mentions.merge_request_id = merge_requests.id)
           Heap Fetches: 0
           Buffers: shared hit=4296 read=6
           I/O Timings: read=20.314
  
  Time: 3.025 s
    - planning: 4.435 ms
    - execution: 3.021 s
      - I/O read: 2.937 s
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 4345 (~33.90 MiB) from the buffer pool
    - reads: 1360 (~10.60 MiB) from the OS file cache, including disk I/O
    - dirtied: 5 (~40.00 KiB)
    - writes: 0


  EXPLAIN SELECT "merge_requests".* FROM "merge_requests"
  LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id
  WHERE ((description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL)
  AND "merge_requests"."id" BETWEEN 47700000 AND 47800000


  Nested Loop Anti Join  (cost=0.86..3729.09 rows=2002 width=732) (actual time=13.497..2573.046 rows=533 loops=1)
     Buffers: shared hit=4286 read=1429 dirtied=1
     I/O Timings: read=2518.843
     ->  Index Scan using merge_request_mentions_temp_index on public.merge_requests  (cost=0.43..1742.71 rows=2031 width=732) (actual time=6.322..2520.994 rows=1426 loops=1)
           Index Cond: ((merge_requests.id >= 47700000) AND (merge_requests.id <= 47800000))
           Buffers: shared hit=18 read=1411 dirtied=1
           I/O Timings: read=2490.673
     ->  Index Only Scan using merge_request_user_mentions_on_mr_id_and_note_id_index on public.merge_request_user_mentions  (cost=0.43..2.06 rows=3 width=4) (actual time=0.032..0.032 rows=1 loops=1426)
           Index Cond: (merge_request_user_mentions.merge_request_id = merge_requests.id)
           Heap Fetches: 0
           Buffers: shared hit=4268 read=18
           I/O Timings: read=28.170
  
  Time: 2.578 s
    - planning: 4.678 ms
    - execution: 2.573 s
      - I/O read: 2.519 s
      - I/O write: 0.000 ms
  
  Shared buffers:
    - hits: 4286 (~33.50 MiB) from the buffer pool
    - reads: 1429 (~11.20 MiB) from the OS file cache, including disk I/O
    - dirtied: 1 (~8.00 KiB)
    - writes: 0

Explain on attempting to batch insert ~500 records on staging read only DB

Click to see the huge insert SQL


  EXPLAIN INSERT INTO merge_request_user_mentions ("merge_request_id", "note_id", "mentioned_users_ids", "mentioned_projects_ids", "mentioned_groups_ids") VALUES (8, NULL, '{313, 4907, 5256, 6547, 7354, 9315}', NULL, NULL), (9, NULL, '{1539, 2627, 9271, 10838}', NULL, NULL), (13, NULL, '{8837, 9469, 10180}', NULL, NULL), (14, NULL, '{1497, 2912, 7337, 7915, 8818}', NULL, NULL), (15, NULL, '{495, 6152, 7267, 10988, 11233, 11802}', NULL, NULL), (19, NULL, '{3223, 3452, 3498, 6105, 8419, 9201, 10160}', NULL, NULL), (20, NULL, '{271, 898, 1572, 2601, 4836, 5910, 7682, 7822, 11686}', NULL, NULL), (22, NULL, '{1506, 4006, 7074, 8297}', NULL, NULL), (23, NULL, '{3032, 11139, 11575}', NULL, NULL), (28, NULL, '{380, 1611, 1722, 2843, 2916, 4383, 5770, 8106, 10867}', NULL, NULL), (29, NULL, '{1220, 1972, 5616, 9104, 9650, 12111, 12293}', NULL, NULL), (32, NULL, '{4888, 7055, 9818}', NULL, NULL), (35, NULL, '{850, 3228, 4315, 4482, 7383, 8246, 11705}', NULL, NULL), (36, NULL, '{20, 24}', NULL, NULL), (40, NULL, '{1806, 3386, 5981, 9912, 11021}', NULL, NULL), (45, NULL, '{666, 5925, 9240, 9278, 10448, 11153}', NULL, NULL), (48, NULL, '{1}', NULL, NULL), (51, NULL, '{3265, 3279, 5788, 7536, 7554, 10255, 10582}', NULL, NULL), (52, NULL, '{765, 791, 3175, 3327, 6553, 6851, 10972, 11922}', NULL, NULL), (55, NULL, '{329, 1006, 2421, 5870, 7052, 12403}', NULL, NULL), (56, NULL, '{1917, 2020, 8144, 9842, 11749}', NULL, NULL), (57, NULL, '{2784, 2982, 3330, 5258, 7121, 12128}', NULL, NULL), (59, NULL, '{3366, 5039, 7144, 7200, 10197}', NULL, NULL), (65, NULL, '{1139, 2986, 8115, 8950, 9964}', NULL, NULL), (67, NULL, '{2494, 3523, 4039, 5559, 6065, 8648, 9038, 9177}', NULL, NULL), (68, NULL, '{1572, 2627, 7813, 10560, 11719, 12041}', NULL, NULL), (70, NULL, '{2911, 4477, 6272, 6361, 6381, 12096, 12148}', NULL, NULL), (72, NULL, '{163, 2517, 3295, 4145, 4318, 8552, 9289}', NULL, NULL), (73, NULL, '{3687, 4330, 5247, 8852, 9586, 10805, 11130, 11321}', NULL, NULL), (74, NULL, '{2130, 2715, 5858, 6421, 8345}', NULL, NULL), (75, NULL, '{3556, 6233, 11434}', NULL, NULL), (83, NULL, '{2233, 3090}', NULL, NULL), (86, NULL, '{189, 2366, 4703, 9067, 12097}', NULL, NULL), (92, NULL, '{1243, 2771, 5886, 6375, 6482, 7240, 8129, 10157, 10952}', NULL, NULL), (95, NULL, '{2662, 3285, 6372, 9104, 10143, 11545}', NULL, NULL), (99, NULL, '{5325, 5972, 10096, 11586}', NULL, NULL), (100, NULL, '{9149}', NULL, NULL), (101, NULL, '{63, 1486, 2720, 2796, 2938, 3303, 4215, 11580, 12189}', NULL, NULL), (102, NULL, '{1692, 2581, 3832, 4113, 4286, 4965, 7296, 8193, 9910}', NULL, NULL), (104, NULL, '{2020, 3486, 4120, 5837, 9276, 9547, 11665}', NULL, NULL), (108, NULL, '{2201, 5722, 7886, 9138, 12099}', NULL, NULL), (113, NULL, '{457, 2014, 6465, 8721}', NULL, NULL), (118, NULL, '{512, 1301, 2990, 5085, 7939, 9425, 10782, 10832, 11625}', NULL, NULL), (121, NULL, '{862, 2534, 9722, 9961}', NULL, NULL), (123, NULL, '{3178, 4975, 7092, 9842, 11245}', NULL, NULL), (126, NULL, '{3040, 3766, 4450, 7448}', NULL, NULL), (127, NULL, '{737, 3168, 4562, 4900, 6438, 7131, 10066}', NULL, NULL), (131, NULL, '{3000, 3820, 7447, 9159, 9442, 12096, 12422}', NULL, NULL), (132, NULL, '{877, 1449, 4368, 5332, 8677, 8746, 8923}', NULL, NULL), (135, NULL, '{58, 242, 295, 9900, 12087}', NULL, NULL), (137, NULL, '{576, 1762, 4922, 5994, 7571}', NULL, NULL), (141, NULL, '{89, 571, 1710, 3874, 4083, 6499, 11399}', NULL, NULL), (143, NULL, '{92, 216, 2509, 3573, 5980, 6991}', NULL, NULL), (151, NULL, '{3136, 3864, 4671, 5922, 12017}', NULL, NULL), (155, NULL, '{302, 1470, 1897, 3759, 6562, 9635}', NULL, NULL), (156, NULL, '{825, 2119, 3208, 4260, 4303, 5311, 11680, 11860}', NULL, NULL), (157, NULL, '{1659, 1874, 9893, 9989}', NULL, NULL), (159, NULL, '{947, 2341, 2520, 3954, 10102, 10282}', NULL, NULL), (162, NULL, '{967, 2041, 5563, 5907, 8401}', NULL, NULL), (163, NULL, '{6003, 9911, 12368}', NULL, NULL), (164, NULL, '{6356, 10981}', NULL, NULL), (172, NULL, '{2201, 2924, 10746, 11875}', NULL, NULL), (175, NULL, '{1193, 3630, 3972, 6105, 6206, 10745, 10796, 11058, 11649}', NULL, NULL), (177, NULL, '{2272, 3241, 4271, 7335, 9996, 10947, 11371}', NULL, NULL), (179, NULL, '{2550, 5298, 6555, 6937, 7967, 9973}', NULL, NULL), (181, NULL, '{3298, 7872, 10104}', NULL, NULL), (182, NULL, '{1292, 2455, 4825, 5441, 9202, 10007, 10314, 10495, 11898}', NULL, NULL), (191, NULL, '{734, 1979, 2428, 3865, 5922, 8172, 8464, 10346, 11068, 11870}', NULL, NULL), (194, NULL, '{1240, 2390, 3846, 3874, 4928, 5866, 7143, 12173}', NULL, NULL), (195, NULL, '{49, 802, 1394, 4566, 12008}', NULL, NULL), (196, NULL, '{566, 2391, 3421, 3557, 6126, 9070, 11593}', NULL, NULL), (198, NULL, '{396, 1333, 3458, 6831, 7751, 9611, 10118}', NULL, NULL), (199, NULL, '{1546, 2367, 5112, 5339, 8070, 10136}', NULL, NULL), (201, NULL, '{1873, 3156, 5433, 6959, 8169, 9289, 11925}', NULL, NULL), (202, NULL, '{655, 4493, 7293, 7505, 12107, 12430}', NULL, NULL), (204, NULL, '{3686, 5551, 6465, 7291, 9049}', NULL, NULL), (206, NULL, '{318, 1376, 4220, 5550, 6333, 6585, 9553, 11465}', NULL, NULL), (207, NULL, '{311, 2240, 4631, 9044, 11012}', NULL, NULL), (209, NULL, '{4416, 4520, 5418, 8567, 9869}', NULL, NULL), (210, NULL, '{143, 1510, 2353, 5225, 8400}', NULL, NULL), (212, NULL, '{2769, 4182, 4383, 10092, 11390}', NULL, NULL), (216, NULL, '{905, 1278, 1987, 3807, 9582, 11475, 11615, 11984}', NULL, NULL), (219, NULL, '{1322, 1623, 1768, 2037, 3189, 6866, 10709}', NULL, NULL), (222, NULL, '{1998, 10718, 11012}', NULL, NULL), (223, NULL, '{251, 469, 9874}', NULL, NULL), (225, NULL, '{1806, 3015, 6156, 8533, 11131}', NULL, NULL), (226, NULL, '{5060, 8490, 9483, 12266, 12347}', NULL, NULL), (229, NULL, '{2693, 2796, 3618, 4668, 4942, 6273, 9435, 11221}', NULL, NULL), (234, NULL, '{686, 1494, 3654, 5240, 5413, 6240, 6352, 6441, 8972, 9744, 10736}', NULL, NULL), (237, NULL, '{136, 2325, 2447, 4019, 4772, 4968, 5525, 6909, 8150}', NULL, NULL), (238, NULL, '{697, 939, 4275, 7053, 9949, 10915}', NULL, NULL), (240, NULL, '{202, 676, 8493, 9242, 11237, 11956}', NULL, NULL), (246, NULL, '{1442, 8944, 10242, 11503}', NULL, NULL), (247, NULL, '{2360, 4650, 5402}', NULL, NULL), (249, NULL, '{2803, 6394, 7678, 8350, 9969, 11590}', NULL, NULL), (251, NULL, '{1273, 2624, 4513, 4777, 6606, 8440}', NULL, NULL), (252, NULL, '{64, 4429, 6187, 7100, 11301, 12399}', NULL, NULL), (253, NULL, '{5513, 6479, 7681, 8193, 8456, 12211}', NULL, NULL), (254, NULL, '{1866, 3738, 4482, 7876}', NULL, NULL), (256, NULL, '{915, 3883, 10710, 11227}', NULL, NULL), (258, NULL, '{43, 1593, 3745, 5201, 5489, 7233, 8080, 8910}', NULL, NULL), (259, NULL, '{1123, 3195, 8439, 9332, 11801}', NULL, NULL), (260, NULL, '{261, 881, 995, 3704, 5333, 6254, 8632, 9823}', NULL, NULL), (263, NULL, '{5522, 10262, 11897}', NULL, NULL), (270, NULL, '{6263, 7017, 7309, 7735, 8329, 9465, 10855}', NULL, NULL), (272, NULL, '{1095, 4063, 4232, 9017}', NULL, NULL), (273, NULL, '{6256, 6625, 6933, 10213, 11372}', NULL, NULL), (274, NULL, '{634, 2875, 5027, 8686, 10160}', NULL, NULL), (279, NULL, '{5899, 7074, 7752, 8722, 9350, 9982, 11622}', NULL, NULL), (280, NULL, '{75, 3853, 4623, 4905, 7512, 11183, 11311}', NULL, NULL), (281, NULL, '{264, 1768, 5008, 5587, 6009, 9114, 12135}', NULL, NULL), (283, NULL, '{857, 3667, 6317, 7454, 8866, 10134}', NULL, NULL), (285, NULL, '{198, 721, 1307, 2619, 3954, 6469, 10119}', NULL, NULL), (288, NULL, '{1170, 3449, 3934, 6463, 10428, 10866, 10951}', NULL, NULL), (291, NULL, '{3687, 6831, 7491, 10745}', NULL, NULL), (292, NULL, '{518, 4703, 5034, 5487, 6993, 8422, 10863, 11235}', NULL, NULL), (294, NULL, '{504, 3004, 7034, 10683, 11952, 11993, 12314}', NULL, NULL), (298, NULL, '{3641, 4939, 6560, 9685, 11344}', NULL, NULL), (299, NULL, '{657, 2462, 6533, 9135}', NULL, NULL), (300, NULL, '{462, 1761, 4436, 4874, 6503, 6542, 8268, 12181}', NULL, NULL), (303, NULL, '{870, 4797, 4902, 6440, 6585, 6800, 8834}', NULL, NULL), (304, NULL, '{710, 2574, 3183, 4312, 5532, 5736, 10998}', NULL, NULL), (307, NULL, '{551, 3095, 3605, 6535, 7188, 8666, 9509}', NULL, NULL), (308, NULL, '{2484, 2924, 3330, 5964, 10787, 11391}', NULL, NULL), (311, NULL, '{1280, 2949, 4161, 8924, 9225, 10233}', NULL, NULL), (315, NULL, '{1877, 5556, 7499, 9293, 9825}', NULL, NULL), (319, NULL, '{882, 1037, 1145, 3032, 3563, 3582, 4339, 8457, 9549, 12044}', NULL, NULL), (322, NULL, '{1121, 5469, 12350}', NULL, NULL), (328, NULL, '{2594, 8728, 9529}', NULL, NULL), (329, NULL, '{1284, 2842, 5338, 6556, 6907, 9124}', NULL, NULL), (330, NULL, '{2633, 5608, 8957}', NULL, NULL), (332, NULL, '{498, 3628, 5708, 5849, 8815, 9867, 10489, 10827, 12161}', NULL, NULL), (333, NULL, '{334, 1009, 1980, 3196, 3240, 3304, 10740, 12209}', NULL, NULL), (339, NULL, '{4172, 5528, 8498, 8797, 9460, 10448}', NULL, NULL), (341, NULL, '{103, 4382, 6141}', NULL, NULL), (342, NULL, '{1506, 1711, 6370, 9298, 9956, 10633}', NULL, NULL), (343, NULL, '{4343, 6040, 8276, 9244, 9634, 10125, 10733}', NULL, NULL), (345, NULL, '{436, 485, 956, 3447, 5086, 8958, 11509, 12284}', NULL, NULL), (349, NULL, '{724, 1201, 2530, 6848, 10746, 10986}', NULL, NULL), (358, NULL, '{1257, 4183, 4621, 4880, 5102, 5932, 12110}', NULL, NULL), (361, NULL, '{708, 2458}', NULL, NULL), (362, NULL, '{3737, 4134, 4849, 10975}', NULL, NULL), (366, NULL, '{1370, 6289, 8167, 9134, 9751, 11083}', NULL, NULL), (368, NULL, '{65, 201, 990, 8608, 9163, 11688}', NULL, NULL), (370, NULL, '{5024, 8835}', NULL, NULL), (372, NULL, '{3210, 3630, 6958, 7305, 11234, 11426, 11890, 11932}', NULL, NULL), (374, NULL, '{438, 750, 4157, 5836, 6429, 6945}', NULL, NULL), (377, NULL, '{1942, 2715, 5501, 5553, 6801, 10813}', NULL, NULL), (379, NULL, '{3490, 3509, 8412, 10671, 11728, 12325}', NULL, NULL), (380, NULL, '{1918, 5498, 6092, 6914, 7009, 7186, 10547}', NULL, NULL), (385, NULL, '{492, 2387, 5231, 8919}', NULL, NULL), (387, NULL, '{1426, 1675, 2585, 3394, 3916, 4623, 7365, 9689, 10181, 11312, 12336}', NULL, NULL), (389, NULL, '{1359, 2470, 2777}', NULL, NULL), (391, NULL, '{2832, 4239, 7056, 8640, 8985, 9828}', NULL, NULL), (393, NULL, '{1080, 6962, 7003, 8483}', NULL, NULL), (396, NULL, '{1070, 1180, 1279, 4262, 5564, 6440, 8578, 9674, 12267}', NULL, NULL), (397, NULL, '{2975, 6979, 7086, 11310, 11547}', NULL, NULL), (399, NULL, '{5197, 10307, 11929}', NULL, NULL), (402, NULL, '{1651, 5760, 5833, 7404, 9173, 10381, 11811}', NULL, NULL), (403, NULL, '{3658, 6524, 8675, 8783}', NULL, NULL), (404, NULL, '{919, 2922, 3084, 3239, 4076, 5846, 8381, 11082}', NULL, NULL), (408, NULL, '{3586, 4969, 5362, 5980, 8335, 11323}', NULL, NULL), (409, NULL, '{5114, 5227, 6986, 7071, 7836, 8474, 11018, 11737}', NULL, NULL), (411, NULL, '{518, 3479, 5988, 7076, 9421}', NULL, NULL), (412, NULL, '{1780, 6149, 7557, 8277, 9122, 9866, 11490, 11679}', NULL, NULL), (413, NULL, '{2845, 3701, 4296, 6622, 7869, 10017}', NULL, NULL), (421, NULL, '{3694, 4918, 5445, 6125, 7527, 8102, 8353, 10033, 11623, 12283}', NULL, NULL), (423, NULL, '{1731, 3906, 7053, 7135, 9339, 11842}', NULL, NULL), (428, NULL, '{2444, 3656, 5664, 7977, 9795, 10706, 11105}', NULL, NULL), (429, NULL, '{692, 4055, 4219, 8741, 9386, 10400, 11312}', NULL, NULL), (431, NULL, '{428, 1312, 1910, 6182, 10188, 10492}', NULL, NULL), (432, NULL, '{2715, 3109, 3683, 5226, 5346, 6909, 7952, 8474, 10948}', NULL, NULL), (433, NULL, '{3195, 3620, 3950, 4509, 9716, 11988}', NULL, NULL), (434, NULL, '{3157, 11447, 11556, 12187}', NULL, NULL), (435, NULL, '{990, 2181, 3835, 4901, 5275, 5308, 9118, 10087}', NULL, NULL), (436, NULL, '{5012, 6939, 7562, 7734, 8480, 10238}', NULL, NULL), (437, NULL, '{6558, 11881}', NULL, NULL), (438, NULL, '{2516, 4932, 5243, 6723}', NULL, NULL), (439, NULL, '{1972, 3135, 4026, 4469, 5858, 8076, 10628}', NULL, NULL), (440, NULL, '{1081, 6076, 7820, 8739, 12154}', NULL, NULL), (441, NULL, '{1732, 2043, 3670, 4000, 5697, 6506, 7626}', NULL, NULL), (445, NULL, '{1562, 4426, 4663, 6488, 10309}', NULL, NULL), (446, NULL, '{741, 2668, 3292, 3617, 6514, 8581, 10523, 11059}', NULL, NULL), (448, NULL, '{6360, 10464, 11588, 11687}', NULL, NULL), (451, NULL, '{333, 5774, 6293, 9743, 9847, 11373}', NULL, NULL), (455, NULL, '{1908, 4084, 4759, 6171, 8298, 11516}', NULL, NULL), (457, NULL, '{2762, 3369, 3610, 6202, 6782, 8861}', NULL, NULL), (461, NULL, '{833, 1526, 4503, 5316}', NULL, NULL), (462, NULL, '{218, 597, 704, 3530, 5622, 6779, 8127, 9729}', NULL, NULL), (464, NULL, '{1640, 1780, 2622, 6795, 6850, 7474, 7864, 8164, 11878}', NULL, NULL), (467, NULL, '{100, 524, 4458, 8131, 12101}', NULL, NULL), (472, NULL, '{2660, 4108, 9308}', NULL, NULL), (473, NULL, '{4044, 4619, 5703, 8549}', NULL, NULL), (474, NULL, '{3124, 4444, 4592, 5381, 5988, 8873, 9882, 9885, 10461}', NULL, NULL), (475, NULL, '{968, 5985, 6815, 7025, 7054, 7912, 11048}', NULL, NULL), (480, NULL, '{327, 7855, 8006, 8374, 9379, 10065}', NULL, NULL), (483, NULL, '{241, 3270, 3811, 4140, 4610, 6244, 7186, 7925, 10052}', NULL, NULL), (485, NULL, '{314, 1554, 1725, 2543, 5230, 5349, 6716, 11488}', NULL, NULL), (486, NULL, '{297, 7001, 7341, 8990, 12116}', NULL, NULL), (491, NULL, '{3513, 4362, 6980, 8454, 10785, 10950}', NULL, NULL), (492, NULL, '{2236, 2869, 8144, 10339, 12262}', NULL, NULL), (495, NULL, '{1158, 3092, 3677, 4657, 4942, 5380, 9626}', NULL, NULL), (499, NULL, '{1878, 3115, 3152, 7708}', NULL, NULL), (501, NULL, '{58, 1407, 6222, 7655, 9778}', NULL, NULL), (502, NULL, '{521, 1442, 3189, 5126, 5676, 8934}', NULL, NULL), (503, NULL, '{3085, 7653, 7759}', NULL, NULL), (504, NULL, '{1856, 5892, 7952}', NULL, NULL), (506, NULL, '{1063, 3243, 4377, 11939}', NULL, NULL), (507, NULL, '{171, 1349, 2696, 6367, 6502, 6713, 9950}', NULL, NULL), (509, NULL, '{14, 82, 1002, 1985, 3922}', NULL, NULL), (511, NULL, '{1339, 7871, 10243, 11037}', NULL, NULL), (515, NULL, '{3314, 3571, 4163, 6766, 8803, 9068, 11259}', NULL, NULL), (516, NULL, '{4370, 5043, 6648, 9067, 10952}', NULL, NULL), (517, NULL, '{584, 1798, 3764, 6786, 6822, 8762}', NULL, NULL), (518, NULL, '{970, 4534, 5216, 10768, 10780, 11409}', NULL, NULL), (520, NULL, '{219, 691, 1221, 2499, 4187, 5594, 9637, 11893}', NULL, NULL), (521, NULL, '{473, 868, 2698, 4439, 5149, 5979, 6357, 9341}', NULL, NULL), (524, NULL, '{1376, 2247, 2605, 6583}', NULL, NULL), (525, NULL, '{7198, 9238}', NULL, NULL), (527, NULL, '{205, 1235, 6045, 10950, 12197}', NULL, NULL), (528, NULL, '{1009, 4148, 4514, 9970, 12432}', NULL, NULL), (530, NULL, '{1419, 2219}', NULL, NULL), (534, NULL, '{3104, 4281, 7363, 11583}', NULL, NULL), (537, NULL, '{1693, 5843, 5913, 9825, 12186}', NULL, NULL), (538, NULL, '{2300, 5060, 8090, 9329, 9911, 11580}', NULL, NULL), (544, NULL, '{1648, 1656, 4619, 5722, 8442, 9757, 10702}', NULL, NULL), (545, NULL, '{977, 2580, 5480, 8354, 8563, 11875}', NULL, NULL), (547, NULL, '{1278, 3717, 5052, 8213, 8843, 11762}', NULL, NULL), (549, NULL, '{1125, 1270, 1992, 3641, 4129, 6827, 9115, 11213}', NULL, NULL), (552, NULL, '{3785, 12127}', NULL, NULL), (557, NULL, '{1988, 3141, 3236, 6352, 8001, 8816, 8883}', NULL, NULL), (559, NULL, '{303, 2397, 2834, 4888, 6683, 6936, 10022, 10598}', NULL, NULL), (564, NULL, '{1943, 3180, 4429, 6268, 7369, 11343}', NULL, NULL), (566, NULL, '{1773, 4439, 4495, 4982, 7918, 10944, 11151}', NULL, NULL), (567, NULL, '{2103, 2845, 6105, 6782, 8353, 8605}', NULL, NULL), (568, NULL, '{2936, 4048, 9199}', NULL, NULL), (569, NULL, '{3989, 4246, 5705, 7390, 7581, 8332, 9241, 10694}', NULL, NULL), (572, NULL, '{1091, 1579, 4355, 8583, 9147, 9473, 10459}', NULL, NULL), (573, NULL, '{2438, 3995, 6060, 11424}', NULL, NULL), (574, NULL, '{697, 3770, 4169, 5416, 5586, 5852, 7074, 9650, 9967}', NULL, NULL), (580, NULL, '{242, 1505, 1539, 2146, 8394, 11209, 12355}', NULL, NULL), (581, NULL, '{993, 3586, 3679, 3792, 6471, 8743, 9627, 11725}', NULL, NULL), (585, NULL, '{3117, 3779, 4003, 4527, 7474, 10173, 10662, 12362}', NULL, NULL), (586, NULL, '{422, 490, 2905, 4879, 12084}', NULL, NULL), (587, NULL, '{11635}', NULL, NULL), (589, NULL, '{2088, 8053, 8107}', NULL, NULL), (590, NULL, '{1531, 6004, 6233, 8944}', NULL, NULL), (591, NULL, '{936, 2149, 7483, 9542, 11871}', NULL, NULL), (592, NULL, '{5998, 6721, 10064, 11658}', NULL, NULL), (593, NULL, '{7739, 8076, 8172, 9863}', NULL, NULL), (594, NULL, '{665, 1194, 2381, 4077, 6367, 6564, 10644, 11108}', NULL, NULL), (595, NULL, '{119, 202, 5854, 6563, 9794, 12436}', NULL, NULL), (600, NULL, '{3672, 6941, 8651}', NULL, NULL), (604, NULL, '{862, 1262, 2306, 3001, 6824}', NULL, NULL), (605, NULL, '{906, 5001, 7751, 9099}', NULL, NULL), (606, NULL, '{1446, 2819, 3373, 7481, 7521, 10550}', NULL, NULL), (608, NULL, '{358, 1069, 1883, 2516, 4335, 5525, 9963}', NULL, NULL), (609, NULL, '{955, 5072, 6323, 10316, 12142}', NULL, NULL), (610, NULL, '{1741, 4572, 5781, 6514, 9291, 11274}', NULL, NULL), (611, NULL, '{607, 836, 4805, 9169, 11580}', NULL, NULL), (616, NULL, '{2359, 6544, 8404, 9050, 10923}', NULL, NULL), (618, NULL, '{447, 1044, 1832, 2360, 9874}', NULL, NULL), (621, NULL, '{2286, 2666, 3591, 8692, 8827, 11107, 12335}', NULL, NULL), (622, NULL, '{3822, 4347, 5449, 7982, 8203, 11312, 11717, 12274}', NULL, NULL), (623, NULL, '{218, 3033, 3508, 3676, 4737, 6671, 7359, 9373}', NULL, NULL), (624, NULL, '{705, 4289, 4313, 7804, 8027, 9604, 10571}', NULL, NULL), (625, NULL, '{4454, 4802, 4924, 6412, 7161, 8267, 11074}', NULL, NULL), (626, NULL, '{2572, 2769, 3634, 4230, 8390, 9168}', NULL, NULL), (629, NULL, '{2857, 3576, 4026, 9024, 9279, 11370, 11569}', NULL, NULL), (634, NULL, '{2131, 2601, 6304, 6836, 7695, 8458, 9930, 11589}', NULL, NULL), (638, NULL, '{692, 3516, 4072, 8288, 11288}', NULL, NULL), (641, NULL, '{882, 2204, 2976, 4517, 9456}', NULL, NULL), (642, NULL, '{10205, 12371}', NULL, NULL), (643, NULL, '{264, 6375}', NULL, NULL), (644, NULL, '{500, 831, 1173, 4973, 5077, 12063}', NULL, NULL), (646, NULL, '{2489, 8873, 9023, 9434, 9977, 11056, 12401}', NULL, NULL), (653, NULL, '{1640, 3443, 4481, 5543, 10284, 12405}', NULL, NULL), (655, NULL, '{2243, 2647, 3094, 3733, 8275, 8593, 9091, 11284}', NULL, NULL), (656, NULL, '{1073, 3718, 5135}', NULL, NULL), (658, NULL, '{2616, 4758, 12067}', NULL, NULL), (659, NULL, '{266, 3618, 4317, 10860, 11881, 12049}', NULL, NULL), (661, NULL, '{1341, 2045, 4487, 5777, 6349, 10883}', NULL, NULL), (663, NULL, '{684, 694, 5116, 7661, 7823, 9720, 10359}', NULL, NULL), (664, NULL, '{1750, 3858, 4295, 4296, 7027, 8781}', NULL, NULL), (666, NULL, '{5742, 8549, 9372, 10359, 10931}', NULL, NULL), (671, NULL, '{1198, 3652, 3946, 5120, 10457}', NULL, NULL), (672, NULL, '{621, 921, 2391, 3286, 6573, 7124, 7324, 10290}', NULL, NULL), (673, NULL, '{2869, 6167, 6413, 6825, 11864}', NULL, NULL), (675, NULL, '{3407, 6946, 7120, 8331, 12054}', NULL, NULL), (676, NULL, '{997, 3104, 8437, 10870, 10989, 11342}', NULL, NULL), (678, NULL, '{847, 2093, 5214, 11668, 12344}', NULL, NULL), (679, NULL, '{295, 2177, 5481, 6321, 9741, 10361}', NULL, NULL), (682, NULL, '{1297, 2111, 2892, 3605, 3888, 7981}', NULL, NULL), (683, NULL, '{122, 2353, 3937, 6329, 6964, 7460, 8770}', NULL, NULL), (685, NULL, '{1673, 3715, 5368, 5484, 10274, 11570}', NULL, NULL), (686, NULL, '{452, 1504, 4582}', NULL, NULL), (687, NULL, '{893, 2336, 3580, 4151, 9635}', NULL, NULL), (688, NULL, '{1023, 8637, 11577}', NULL, NULL), (689, NULL, '{1671, 1915, 2372, 4102, 5296, 7574, 7762, 11362}', NULL, NULL), (692, NULL, '{1148, 2892, 6455, 8006, 11959}', NULL, NULL), (694, NULL, '{252, 1354, 5830, 8796, 9829, 9839}', NULL, NULL), (695, NULL, '{2845, 4619, 4864, 10203, 10308}', NULL, NULL), (697, NULL, '{4892, 5942, 8224, 9373, 11019}', NULL, NULL), (698, NULL, '{4201, 6949, 9291, 10918}', NULL, NULL), (699, NULL, '{742, 2682, 7536, 8270, 9382, 10369, 10639}', NULL, NULL), (700, NULL, '{2187, 6302}', NULL, NULL), (701, NULL, '{594, 3791, 5505, 6869, 7674, 11848, 11902}', NULL, NULL), (702, NULL, '{1810, 2221, 7280, 9146}', NULL, NULL), (703, NULL, '{1217, 2509, 2635, 7664, 10702}', NULL, NULL), (706, NULL, '{2667, 2708, 4544, 5826, 6409, 8349, 12243}', NULL, NULL), (709, NULL, '{4425, 5942, 6268, 7147, 9599, 10713}', NULL, NULL), (716, NULL, '{7, 828, 4129, 7514, 9921, 11194}', NULL, NULL), (717, NULL, '{4239, 6349, 8698, 8992, 10175, 10671}', NULL, NULL), (718, NULL, '{92, 902, 1338, 2685, 3203, 9853, 11788}', NULL, NULL), (720, NULL, '{374, 2835, 3287, 4676, 6771, 10297}', NULL, NULL), (721, NULL, '{59, 1113, 3365, 6321, 10018, 11405}', NULL, NULL), (722, NULL, '{1631, 2221, 2346, 3443, 5655, 5758, 8366, 10931, 11006, 11136}', NULL, NULL), (726, NULL, '{2541, 3118, 4236, 5393, 7016, 9101, 10576}', NULL, NULL), (728, NULL, '{66, 2005, 2396, 6286, 6359, 6979, 7160, 12391}', NULL, NULL), (729, NULL, '{2361, 3632, 4756, 5947, 10896}', NULL, NULL), (731, NULL, '{981, 2291, 3790, 5597, 5874, 6225, 11022, 12200}', NULL, NULL), (733, NULL, '{1059, 1485, 5487, 5750, 8654, 11633}', NULL, NULL), (738, NULL, '{2933, 3173, 7123, 9104}', NULL, NULL), (740, NULL, '{1224, 3979, 10354, 10961}', NULL, NULL), (742, NULL, '{4205, 4313, 6978, 9582}', NULL, NULL), (743, NULL, '{3084, 3670, 5573, 8175, 11339, 11429}', NULL, NULL), (748, NULL, '{77, 3621, 6185, 7003, 7268, 8872, 10705, 11999}', NULL, NULL), (752, NULL, '{1152, 5850, 6113, 6253, 6513, 6961, 9696}', NULL, NULL), (754, NULL, '{615, 765, 1333, 4667, 5334, 6528, 7564, 8222, 9888}', NULL, NULL), (755, NULL, '{4866, 5021, 5296, 8786, 10858, 12405}', NULL, NULL), (756, NULL, '{1906, 10070}', NULL, NULL), (757, NULL, '{3669, 6979, 7921, 10600}', NULL, NULL), (760, NULL, '{465, 954, 1007, 1491, 1593, 3579, 6888, 10342, 10590, 10907}', NULL, NULL), (761, NULL, '{278, 1104, 4991, 11917}', NULL, NULL), (763, NULL, '{1305, 2286, 4329, 6512, 7734, 9164, 12428}', NULL, NULL), (764, NULL, '{124, 494, 4350, 6891, 8611, 11251}', NULL, NULL), (767, NULL, '{1356, 1424, 2808, 7141, 7945, 7997, 9672}', NULL, NULL), (768, NULL, '{1447, 2543, 3317, 4217, 4555, 5150, 9782, 12080}', NULL, NULL), (770, NULL, '{2047, 2721, 3921, 3988, 8854, 11125}', NULL, NULL), (772, NULL, '{3993, 9061}', NULL, NULL), (773, NULL, '{1225, 2366, 2644, 2974, 4121, 4710, 6062, 7360, 12025}', NULL, NULL), (774, NULL, '{2897, 3578, 6464, 6660, 6934, 8730, 8886}', NULL, NULL), (775, NULL, '{1444, 5809, 6501}', NULL, NULL), (777, NULL, '{2797, 3489, 5363, 5933, 7334, 8716, 10840, 12246}', NULL, NULL), (780, NULL, '{3430, 4779, 7853, 8409, 8587, 9154}', NULL, NULL), (784, NULL, '{3857, 10652, 11304, 12182}', NULL, NULL), (786, NULL, '{1789, 4034, 6800, 9343}', NULL, NULL), (787, NULL, '{1845, 4376, 5963, 10390}', NULL, NULL), (788, NULL, '{893, 2531, 2707, 8286, 9656, 11013, 11026}', NULL, NULL), (793, NULL, '{2806, 3268, 4544, 6021, 8257, 10089, 10268, 10298, 12327}', NULL, NULL), (795, NULL, '{6613, 6703, 9301}', NULL, NULL), (797, NULL, '{1644, 2858, 3828, 6565, 6586, 10736, 12163}', NULL, NULL), (798, NULL, '{76, 81, 279, 989, 6192, 7607, 7640, 8584}', NULL, NULL), (799, NULL, '{878, 6450, 9407, 11608, 11978}', NULL, NULL), (803, NULL, '{1352, 7648, 7671, 9292, 9397, 9485, 10447}', NULL, NULL), (805, NULL, '{3826, 4815, 5265, 8206, 8840}', NULL, NULL), (807, NULL, '{1747, 2174, 9636, 9771}', NULL, NULL), (810, NULL, '{542, 3685, 4421, 4498, 5332, 7330, 7385, 11255, 11470, 12212}', NULL, NULL), (811, NULL, '{4172, 5267, 8942, 9272, 10231}', NULL, NULL), (812, NULL, '{2048, 2531, 6757, 7490, 7607, 11143}', NULL, NULL), (813, NULL, '{2760, 5047, 6656, 7876, 9440, 11998}', NULL, NULL), (816, NULL, '{2949, 3937, 4437, 4794, 6180, 6770, 11301}', NULL, NULL), (821, NULL, '{4057, 6862, 7720, 8691}', NULL, NULL), (822, NULL, '{5421, 5561, 7462}', NULL, NULL), (823, NULL, '{1351, 1825, 4067, 5429, 7904, 11397}', NULL, NULL), (824, NULL, '{1943, 4326, 6367, 6682, 7204, 11765}', NULL, NULL), (828, NULL, '{6622, 6917, 9085, 9481, 9792, 10781}', NULL, NULL), (832, NULL, '{1144, 9831, 10744, 11215}', NULL, NULL), (834, NULL, '{1249, 1462, 1607, 7171, 9015, 11141, 12084}', NULL, NULL), (840, NULL, '{315, 3375, 3670, 6020, 11818}', NULL, NULL), (841, NULL, '{398, 5927, 10222, 10965, 11212}', NULL, NULL), (842, NULL, '{826, 6045, 8692, 10028, 11272, 12022}', NULL, NULL), (843, NULL, '{3347, 9418, 9562}', NULL, NULL), (851, NULL, '{3658, 3685, 5333, 5549, 5634, 8778, 9634}', NULL, NULL), (854, NULL, '{2166, 3608, 5214, 7068}', NULL, NULL), (856, NULL, '{473, 837, 5094, 6488, 6952, 10203, 11082, 11279, 11446}', NULL, NULL), (860, NULL, '{631, 1260, 1311, 4712, 5876, 6586, 7228, 8716, 11226, 12194}', NULL, NULL), (865, NULL, '{744, 785, 1220, 1224, 1789, 3663}', NULL, NULL), (866, NULL, '{2010, 4827, 5761, 7060, 9559}', NULL, NULL), (868, NULL, '{42, 2101, 2656, 3910, 7710, 9305, 11952}', NULL, NULL), (869, NULL, '{3540, 3924, 5407, 6736, 8268, 10267, 12190}', NULL, NULL), (876, NULL, '{881, 5074, 6854, 7263, 9772, 11958}', NULL, NULL), (880, NULL, '{1770, 2264, 3925, 4082, 4231, 5985, 12165, 12280}', NULL, NULL), (881, NULL, '{2785, 5003, 5085, 5586, 7423, 11195}', NULL, NULL), (882, NULL, '{570, 2791, 3794, 5780, 9702, 10148}', NULL, NULL), (883, NULL, '{2363, 3224, 4141, 4620, 6086, 12141}', NULL, NULL), (884, NULL, '{1913, 6380, 10820, 11916, 11942}', NULL, NULL), (885, NULL, '{2219, 3829, 4117, 5121, 7566}', NULL, NULL), (886, NULL, '{350, 6143, 7396, 7710, 7810, 9284, 10627, 10673, 12159}', NULL, NULL), (892, NULL, '{3201, 5686, 10476, 10916}', NULL, NULL), (894, NULL, '{2137, 3408, 6396, 8408, 9760, 11602, 12273}', NULL, NULL), (896, NULL, '{1079, 1278, 4301, 5294, 6071, 9392, 10191, 10600}', NULL, NULL), (897, NULL, '{1884, 3229, 3297, 3344, 9737, 11502}', NULL, NULL), (899, NULL, '{6882, 7694, 8990, 10430}', NULL, NULL), (900, NULL, '{2561, 4340, 8006, 8663, 9099, 9300, 11428}', NULL, NULL), (901, NULL, '{1098, 2119, 11366}', NULL, NULL), (903, NULL, '{2644, 6941, 8595, 11543}', NULL, NULL), (905, NULL, '{1245, 2825, 7112, 7352, 10931, 12275}', NULL, NULL), (906, NULL, '{534, 803, 955, 5512, 6712, 8607, 12241}', NULL, NULL), (908, NULL, '{838, 1666, 5811, 6100, 9420, 9751}', NULL, NULL), (914, NULL, '{3521, 3710, 5535, 6380, 8648}', NULL, NULL), (915, NULL, '{556, 1858, 2383, 3506, 7130, 7515, 10440}', NULL, NULL), (921, NULL, '{491, 775, 2245, 3188, 4503, 4638}', NULL, NULL), (924, NULL, '{2912, 3798, 5433, 6307, 6866, 11562}', NULL, NULL), (926, NULL, '{1590, 1768, 2898, 11153}', NULL, NULL), (929, NULL, '{1192, 2004, 2375, 3950, 7288, 9711}', NULL, NULL), (934, NULL, '{5704, 8947, 9404, 9847, 12315}', NULL, NULL), (936, NULL, '{937, 4677, 5380, 7331, 9656}', NULL, NULL), (939, NULL, '{665, 680, 5072, 5920, 7111, 9557, 10750, 11417, 12326}', NULL, NULL), (940, NULL, '{1254, 6151, 9168, 11391}', NULL, NULL), (941, NULL, '{217, 1258, 6972, 7949, 10971, 11642}', NULL, NULL), (945, NULL, '{7345, 8999, 9663, 9733, 12413}', NULL, NULL), (946, NULL, '{2261, 6498, 8141, 9407, 9621, 10207}', NULL, NULL), (949, NULL, '{4642, 4866, 5441, 11539}', NULL, NULL), (951, NULL, '{1053, 2061, 2775, 8760, 10079, 10500}', NULL, NULL), (954, NULL, '{3347, 4247, 4488, 9651, 9806, 12242}', NULL, NULL), (957, NULL, '{10352, 12112}', NULL, NULL), (958, NULL, '{4642, 5663, 5938, 7181, 10924}', NULL, NULL), (959, NULL, '{3022, 3053, 3227, 3594, 3776, 7927, 10468, 11447}', NULL, NULL), (960, NULL, '{1569, 6183, 8164, 11701}', NULL, NULL), (962, NULL, '{474, 3543, 5403, 8323, 8376, 8482, 12276}', NULL, NULL), (966, NULL, '{706, 1617, 4599, 10292, 11859}', NULL, NULL), (967, NULL, '{2561, 3207, 3998, 6708}', NULL, NULL), (968, NULL, '{43, 2546, 4315, 6190, 10946}', NULL, NULL), (969, NULL, '{376, 4154, 4176, 9053, 10945, 11401, 11563}', NULL, NULL), (971, NULL, '{399, 513, 5874, 8384, 9198, 10462, 10801, 10878}', NULL, NULL), (972, NULL, '{4392, 4480, 4768, 9820, 12361}', NULL, NULL), (974, NULL, '{2286, 4783, 6043, 10830, 12407}', NULL, NULL), (977, NULL, '{3093, 3364, 4944, 5756, 6227, 6793, 7627}', NULL, NULL), (979, NULL, '{692, 4694, 6658, 9386}', NULL, NULL), (981, NULL, '{702, 1831, 2253, 2567, 3600, 5513, 7286}', NULL, NULL), (991, NULL, '{2441, 3418, 4488, 6539}', NULL, NULL), (992, NULL, '{1662, 2202, 4981, 5546, 6311, 8448, 9920}', NULL, NULL), (995, NULL, '{2961, 3699, 3948, 4598, 9892}', NULL, NULL), (997, NULL, '{2337, 3058, 6030, 10403}', NULL, NULL), (998, NULL, '{1661, 6718, 7685}', NULL, NULL), (999, NULL, '{9622, 9762, 10279, 11018, 11910}', NULL, NULL), (1002, NULL, '{812, 6437, 9150}', NULL, NULL), (1006, NULL, '{359, 971, 1129, 1310, 1522, 2951, 4453, 4534}', NULL, NULL), (1007, NULL, '{3316, 10610, 11647, 11694}', NULL, NULL), (1010, NULL, '{2434, 4297, 5198, 5962, 9203}', NULL, NULL), (1012, NULL, '{3356, 5549, 7094, 11467, 11571}', NULL, NULL), (1013, NULL, '{1731, 2590, 7282, 9172}', NULL, NULL), (1014, NULL, '{593, 2945, 3098, 3662, 3717, 6827, 7539, 8229}', NULL, NULL), (1017, NULL, '{4704, 10437, 10701, 11836, 12084, 12214}', NULL, NULL), (1018, NULL, '{2313, 3985, 4084}', NULL, NULL), (1021, NULL, '{785, 2476, 3430, 5063, 5466, 8087, 11846}', NULL, NULL), (1022, NULL, '{218, 4404, 5908, 7446}', NULL, NULL), (1024, NULL, '{992, 2576, 5782, 6129, 6771, 6897, 7223, 11838}', NULL, NULL), (1027, NULL, '{4094, 5908, 6284, 7116, 7412, 10260, 10281}', NULL, NULL), (1029, NULL, '{1496, 2442, 3971, 4355, 6340}', NULL, NULL), (1033, NULL, '{430, 4213, 5569, 10268}', NULL, NULL), (1034, NULL, '{721, 4201, 11767}', NULL, NULL), (1037, NULL, '{1268, 2149, 7337, 8330, 8900}', NULL, NULL), (1041, NULL, '{2188, 2399, 5821, 6300, 12400}', NULL, NULL), (1042, NULL, '{483, 4130, 6632, 9319, 9744, 9994, 10369, 11215, 11928}', NULL, NULL), (1043, NULL, '{513, 1068, 1133, 3728, 3925, 4644, 8658, 11289}', NULL, NULL), (1049, NULL, '{5961, 5996, 6799, 8465}', NULL, NULL), (1051, NULL, '{1050, 1667, 1934, 2610, 3106, 4566, 8773, 8961, 12365}', NULL, NULL), (1052, NULL, '{1310, 5439, 6290, 6875, 8294, 10516, 12365, 12397}', NULL, NULL), (1054, NULL, '{2490, 2584, 11219, 11663, 11702}', NULL, NULL), (1056, NULL, '{1723, 4665, 10267, 10850, 10908}', NULL, NULL), (1058, NULL, '{633, 1476, 3246, 3465, 4850, 5063, 6019}', NULL, NULL), (1060, NULL, '{565, 6986, 7872, 7921, 8918, 9056, 11117}', NULL, NULL), (1063, NULL, '{1606, 4485}', NULL, NULL), (1067, NULL, '{269, 677, 4754, 6243, 8103, 9366, 11567}', NULL, NULL), (1068, NULL, '{379, 776, 1590, 2515, 6919, 7075, 8511, 8819}', NULL, NULL), (1074, NULL, '{5227, 5484, 6642, 12244}', NULL, NULL), (1076, NULL, '{988, 1913, 4229, 5916, 7938, 8219}', NULL, NULL), (1077, NULL, '{3237, 5875, 6114, 9245, 10895, 11031, 12127}', NULL, NULL), (1078, NULL, '{815, 7368, 8951, 10589}', NULL, NULL), (1079, NULL, '{1203, 4242, 7223, 9598, 10017, 11042, 11826, 12064}', NULL, NULL), (1082, NULL, '{1942, 6994, 7517, 9152, 11848, 12214}', NULL, NULL), (1094, NULL, '{524, 798, 864, 6004, 8481, 11027, 11195, 11214}', NULL, NULL), (1097, NULL, '{3324, 3333, 5760, 6796, 10694, 12053, 12389}', NULL, NULL), (1098, NULL, '{323, 3437, 4038, 5732, 7440, 9311, 12324}', NULL, NULL), (1099, NULL, '{4391, 6070, 6194, 8493, 8726, 9698}', NULL, NULL), (1100, NULL, '{1098, 2198, 4749, 7028, 7454, 11063}', NULL, NULL), (1103, NULL, '{2426, 3267, 5782, 6419}', NULL, NULL), (1106, NULL, '{7746, 8027, 8547, 12368}', NULL, NULL), (1107, NULL, '{1991, 7225, 9501, 12227}', NULL, NULL), (1110, NULL, '{2424, 7588, 11522, 11871}', NULL, NULL), (1113, NULL, '{167, 2284, 3832, 8023, 11487}', NULL, NULL), (1117, NULL, '{1497, 1889, 2802, 4399, 5992, 6807, 8573, 11191}', NULL, NULL), (1119, NULL, '{1660, 1864, 3623, 3643, 6668, 8378, 8815, 11818}', NULL, NULL), (1120, NULL, '{889, 1037, 5189, 11988, 12206}', NULL, NULL), (1121, NULL, '{885, 1573, 6054, 6961, 7365, 8409, 8933, 9603}', NULL, NULL), (1122, NULL, '{666, 781, 3235, 5116, 5305, 5747}', NULL, NULL), (1123, NULL, '{2056, 2876, 5252}', NULL, NULL), (1124, NULL, '{2257, 5088, 8180, 11857}', NULL, NULL), (1126, NULL, '{541, 2622, 3165, 8523, 8761, 9708}', NULL, NULL), (1127, NULL, '{2922, 3710, 7448, 11543}', NULL, NULL), (1128, NULL, '{2916, 5701}', NULL, NULL), (1129, NULL, '{2102, 4368, 5589, 6258, 7466, 11119}', NULL, NULL), (1131, NULL, '{8204, 8484, 8777, 10454, 11313}', NULL, NULL), (1133, NULL, '{3153, 4714, 5116, 7046, 8263, 9936, 10265}', NULL, NULL), (1136, NULL, '{890, 11107}', NULL, NULL), (1139, NULL, '{1752, 2302, 2737, 9250, 9427, 10547, 11405}', NULL, NULL), (1140, NULL, '{37, 470, 2247, 7715, 8934, 10464}', NULL, NULL), (1141, NULL, '{353, 3374, 4043, 5698, 9169, 11458}', NULL, NULL), (1142, NULL, '{5038, 7682, 8773}', NULL, NULL), (1143, NULL, '{1775, 5937, 7742, 9470}', NULL, NULL) ON CONFLICT DO NOTHING RETURNING id;

Click to see the query plan on insert SQL


   Insert on merge_request_user_mentions  (cost=0.00..7.53 rows=502 width=112)
     Conflict Resolution: NOTHING
     ->  Values Scan on "*VALUES*"  (cost=0.00..7.53 rows=502 width=112)
  (3 rows)

Estimates:

Enqueueing a background jobs

Given the above query plans it looks that it takes under 1s to enqueue 1 background job, which with batches of 100K records that results in ~~500~~ 533 batches of ~1 seconds => ~~~500s~~ 533s => ~9m. So ~9m to enqueue all the background jobs.

Background job runtime estimate

I cannot test such big batches locally, but for a batch of ~1150 merge requests, ~500 of which had mentions the job took ~45s to parse and insert the 500 records.

Click to see the background job log with duration details


2020-06-18_10:45:07.34028 rails-background-jobs : {
"severity": "INFO",
"time": "2020-06-18T10:45:07.339Z",
"class": "BackgroundMigrationWorker",
"args": [
"UserMentions::CreateResourceUserMention",
"["MergeRequest", "LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id", "(description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL", false, 1, 1143]"
],
"retry": 3,
"queue": "background_migration",
"backtrace": true,
"jid": "1fbd732d6350e7eaac7c99cd",
"created_at": "2020-06-18T10:41:19.544Z",
"meta.caller_id": "MigrateAllMergeRequestUserMentionsToDb",
"correlation_id": "87f62ca4cfdf04ffa57a653ce8364758",
"uber-trace-id": "73218164eca3c767:73218164eca3c767:0:1",
"enqueued_at": "2020-06-18T10:44:22.159Z",
"pid": 74701,
"message": "BackgroundMigrationWorker JID-1fbd732d6350e7eaac7c99cd: done: 45.178763 sec",
"job_status": "done",
"scheduling_latency_s": 0.001643,
"gitaly_calls": 5,
"gitaly_duration_s": 0.052518,
"rugged_calls": 109,
"rugged_duration_s": 1.195237,
"redis_calls": 29,
"redis_duration_s": 0.019111,
"redis_read_bytes": 1712,
"redis_write_bytes": 3353,
"redis_cache_calls": 25,
"redis_cache_duration_s": 0.017331,
"redis_cache_read_bytes": 1705,
"redis_cache_write_bytes": 2939,
"redis_queues_calls": 2,
"redis_queues_duration_s": 0.000838,
"redis_queues_read_bytes": 2,
"redis_queues_write_bytes": 189,
"redis_shared_state_calls": 2,
"redis_shared_state_duration_s": 0.000942,
"redis_shared_state_read_bytes": 5,
"redis_shared_state_write_bytes": 225,
"db_count": 6358,
"db_write_count": 1012,
"db_cached_count": 2,
->>>>"duration_s": 45.178763,
"cpu_s": 23.208253,
"completed_at": "2020-06-18T10:45:07.339Z",
->>>>"db_duration_s": 21.094868
}

This is definitely not relevant for production data, but I am not sure how would I be able to test this migration on a production like DB to get more relevant data. This would be important for estimating the delay between background jobs. Currently the delay is 1 minute per 500 matching records to be migrated.

If I extrapolate from that, having ~~620K~~ ~662984 merge requests that contain @ character on production according to #database-lab, we end up with migrating ~~620K~~ ~662984 mentions. If we average that to 500 records/minute we get ~1326mins => ~22h

database-lab: 07-22-2020

explain SELECT "merge_requests".id FROM "merge_requests"   
LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id   
WHERE ((description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL)

Merge Anti Join  (cost=0.86..116917.13 rows=1249764 width=4) (actual time=7.655..28901.928 rows=662984 loops=1)
   Merge Cond: (merge_requests.id = merge_request_user_mentions.merge_request_id)
   Buffers: shared hit=926814 read=23701 dirtied=4180 written=538
   I/O Timings: read=23392.964 write=42.674
   ->  Index Only Scan using merge_request_mentions_temp_index on public.merge_re

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
[-] Tested in all supported browsers
[-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Edited Jul 22, 2020 by Mayra Cabrera

Migrate all merge request user mentions to db table