Skip to content

Add present_on_default_branch to partial_devops_adoption index

Marcos Rocha requested to merge mc_rocha-update-add-vulnerabilities-index-asynchronously-for-project-created-at-default-branch-324858 into master

What does this MR do and why?

Describe in detail what your merge request does and why.

This Merge Request adds the present_on_default_branch attribute to the project_id and created_at index on the vulnerabilities table.

The present_on_default_branch field will be used on most queries, and the vulnerabilities indices should be rearranged.

More context can be found on this comment and in issue #368323 (closed)

This Merge Request is related to issue #368323 (closed)

Migration Up

main: == 20220721141838 UpdateVulnerabilitiesProjectIdCreatedAtIndex: migrating =====
main: -- transaction_open?()
main:    -> 0.0000s
main: -- index_exists?(:vulnerabilities, [:project_id, :created_at, :present_on_default_branch], {:where=>"state != 1", :name=>"idx_vulnerabilities_partial_devops_adoption_and_default_branch", :algorithm=>:concurrently})
main:    -> 0.0137s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0004s
main: -- add_index(:vulnerabilities, [:project_id, :created_at, :present_on_default_branch], {:where=>"state != 1", :name=>"idx_vulnerabilities_partial_devops_adoption_and_default_branch", :algorithm=>:concurrently})
main:    -> 0.0026s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- indexes(:vulnerabilities)
main:    -> 0.0102s
main: -- remove_index(:vulnerabilities, {:algorithm=>:concurrently, :name=>"idx_vulnerabilities_partial_devops_adoption"})
main:    -> 0.0019s
main: == 20220721141838 UpdateVulnerabilitiesProjectIdCreatedAtIndex: migrated (0.0386s)

Migration Down

main: == 20220721141838 UpdateVulnerabilitiesProjectIdCreatedAtIndex: reverting =====
main: -- transaction_open?()
main:    -> 0.0000s
main: -- index_exists?(:vulnerabilities, [:project_id, :created_at], {:where=>"state != 1", :name=>"idx_vulnerabilities_partial_devops_adoption", :algorithm=>:concurrently})
main:    -> 0.0134s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- add_index(:vulnerabilities, [:project_id, :created_at], {:where=>"state != 1", :name=>"idx_vulnerabilities_partial_devops_adoption", :algorithm=>:concurrently})
main:    -> 0.0062s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- indexes(:vulnerabilities)
main:    -> 0.0093s
main: -- remove_index(:vulnerabilities, {:algorithm=>:concurrently, :name=>"idx_vulnerabilities_partial_devops_adoption_and_default_branch"})
main:    -> 0.0030s
main: == 20220721141838 UpdateVulnerabilitiesProjectIdCreatedAtIndex: reverted (0.0424s)

Query explanation

SELECT
    COUNT(*)
FROM
    "projects"
WHERE
    "projects"."id" IN (278964, 11520423, 11574953, 13348998, 13664986, 14097509, 14645774, 14921629, 15477483, 15572444, 15662771, 15704782, 16239404, 16513043, 16700143, 17181705, 17210811, 17743501, 18925373, 18977208, 19042106, 19402176, 19532875, 20401626, 20867992, 21224328, 21483460, 21734866, 21740210, 22133520, 22907418, 23601234, 23944319, 24134396, 24200638, 24763979, 25534115, 26630862, 24693199, 25567144, 13083, 13764, 14022, 14288, 14289, 16648, 19776, 20085, 20086, 20699, 23081, 27468, 27470, 27726, 29286, 32732, 36743, 72724, 74823, 83282, 95156, 98024, 116212, 140724, 143237, 145205, 150440, 227582, 250324, 250833, 278964, 280425, 375711, 387896, 430285, 443787, 444821, 455030, 480929, 554859, 593728, 629054, 629060, 684698, 730448, 734943, 747741, 766015, 818896, 876090, 887372, 928825, 931715, 998792, 1075790, 1120019, 1209837, 1265999, 1329047, 1379171, 1441932, 1470839, 1507906, 1533158, 1777822, 1794617, 1911766, 1990920, 2009901, 2127625, 2317465, 2337675, 2347063, 2383700, 2651596, 2670515, 2694799, 2725567, 2890326, 2903741, 2953390, 3010986, 3010998, 3094319, 3101096, 3305972, 3362933, 3430480, 3466815, 3588247, 3605985, 3631141, 3651684, 3662568, 3662668, 3674569, 3698388, 3871132, 3871556, 3885956, 3885980, 3933206, 3933372, 3991945, 4108541, 4121724, 4149988, 4157550, 4176070, 4186738, 4186742, 4262096, 4341757, 4382108, 4534254, 4542834, 4562632, 4870361, 4921652, 4949400, 5019307, 5261717, 5420758, 5647182, 5981322, 6043225, 6043227, 6092883, 6185025, 6457868, 6466002, 7071551, 7540679, 7602162, 7792567, 7937396, 8343284, 8362868, 8671969, 8987255, 9134345, 9149724, 9359704, 9438583, 9492132, 9780400, 9927571, 10024275, 10614162, 10619765, 10786772, 10947578, 11047304, 11067315, 11080193, 11261137, 11446522, 11997420, 12051824, 12104213, 12212610, 12274094, 12425329, 13015884, 13053017, 13284652, 13368043, 13453461, 13473820, 13683636, 13831684, 13942461, 13997108, 14018700, 14025202, 14071101, 14146123, 14217204, 14290884, 14378900, 14520336, 14694517, 14707715, 14771920, 14785608, 14850988, 15032634, 15158038, 15297693, 15363819, 15420077, 15445353, 15447440, 15461649, 15502498, 15687385, 15815706, 15816248, 15943672, 16110032, 16505341, 16573099, 16603968, 16732333, 16842968, 17039597, 17113313, 17133307, 17318793, 17334694, 17345914, 17429946, 17522813, 17661412, 17930014, 18060348, 18307741, 18307889, 18331927, 18594323, 18594390, 18629149, 18741849, 18776356, 18860383, 18863350, 18893322, 18912995, 18943607, 19031064, 19461377, 19657914, 19861191, 20059805, 20326496, 20468480, 20510065, 20904766, 21250701, 21439066, 21479995, 21565866, 21751536, 21751817, 21808150, 21819002, 21967079, 21967100, 22217400, 22363055, 22795840, 22874770, 23105702, 23156104, 23176166, 23617987, 23627092, 23838512, 23854438, 24467497, 24671550, 24873101, 25031023, 25033712, 25205782, 25312761, 25398815, 25402115, 25416006, 25417540, 25418438, 25837735, 25847700, 25861038, 25981420, 26130408, 26250592, 26576566, 26744820, 27244047, 27490309, 22351703, 9281155, 16302012, 17843116, 15114652, 19722963, 19969997, 20352787, 15065439, 16552062, 16706750, 17192985, 19533665, 20376798, 4359271, 7682093, 14588374, 19310217, 20085049, 8377576, 10071001, 10071132, 19974437, 22438483, 25691329, 17349153, 13017970, 13017990, 13018017, 16488249, 16756957, 18641703, 22414284, 19922160, 13815397, 21324924, 21628863, 23544263, 5677844, 10523039, 14402567, 15962657, 16297627, 19551890, 20842531, 22850480, 24345542, 9184510, 10400718, 11688089, 14404642, 15828516, 19019153, 21764500, 22517790, 15019630, 21635031, 22487050, 23203100, 23344605, 23586678, 10747057, 13142177, 21055520, 18256369, 21055609, 9237774, 12558964, 13017629, 16863027, 17616536, 18256375, 25838334, 25839455, 25840330, 25840667, 27079314, 19043830, 21731412, 21731462, 22046463, 22070439, 22382594, 22408078, 22409005, 22413375, 25944737, 19890267, 23069693, 23069956, 23070140, 23070164, 23070377, 23070395, 23176783, 23176971, 23505725, 23599168, 24302484, 24313398, 24313598, 24317064, 24460210, 24542363, 24544020, 24544330, 25230764, 25298439, 25537666, 25538517, 25560556, 25829938, 26217586, 26217656, 26549908, 26598766, 26638651, 26808701, 26808774, 26882341, 26882878, 26997949, 21055409, 3828396, 6329679, 7453181, 11915984, 14372596, 15393051, 16590122, 17253921, 18439881, 18663049, 18991900, 21525059, 23495190, 21055344, 19800404, 21055767, 20209910, 22215226, 22336999, 22390221, 20110067, 20744986, 15412762, 15499610, 19936368, 13490286, 13672316, 13672681, 13673582, 13673632, 13673686, 13673757, 13673934, 13674085, 13674136, 13675927, 13675979, 13676002, 13676032, 13676096, 13676142, 13676188, 13676249, 13676309, 13676350, 13824926, 16613296, 16622651, 16819118, 16819125, 16819131, 16819134, 16819140, 16819151, 16819248, 17023897, 17043690, 17134833, 17530059, 17821569, 17823896, 18770944, 18789638, 18792504, 19290493, 19290857, 19315261, 20498312, 23508645, 17863546, 19209142, 19245461, 17257555, 21824450, 24339834, 20466430, 17584993, 24470059, 16816732, 17311004, 17433833, 18759998, 18760252, 19265543, 19536993, 20516845, 20747799, 20791019, 20916876, 21573386, 21796485, 22249364, 22566258, 22637103, 23597027, 23690656, 23730491, 23916841, 24300893, 24451242, 24501533, 25158783, 25247374, 26079419, 26584477, 26815390, 27209445, 19843335, 20623750, 22679171, 22734535, 26566881, 27214210, 27219974, 27226337, 27227767, 27229445, 6374580, 6374588, 6374596, 22810929, 22810940, 22811022, 23928229, 17245095, 17990319, 18208929, 18766749, 18976211, 19279270, 19568296, 19655424, 19993243, 20081132, 20630561, 20839085, 21649753, 22847718, 23512541, 23526204, 25969582, 18855526, 27051617, 27051800, 27286937, 27484836, 27484846, 27484871, 24907469, 25216298, 27494514, 26615930, 26615960, 26616051, 26616061, 26616069, 26698878, 26699071, 26699200, 26710947, 27209463, 27220296, 27221011, 27456355, 27456706, 27456810, 27457647, 18899486, 27260823, 27261025, 21573444, 23101765, 23740056, 27213646, 27327788, 18256394, 13954053, 14130725, 16549277, 21486653, 18899500, 22848448, 9172481, 10947320, 12507547, 15486630, 15926775, 20360386, 14392833, 16913647, 18156681, 18875792, 19028203, 19720958, 22130851, 22433156, 23345837, 12138769, 12959103, 13327609, 13437682, 15710326, 16466244, 17966464, 20107267, 25891188, 18406509, 18922546, 18650560, 18766720)
    AND EXISTS (
        SELECT
            "vulnerabilities".*
        FROM
            "vulnerabilities"
        WHERE
            "vulnerabilities"."state" != 1
            AND "vulnerabilities"."created_at" BETWEEN '2022-05-01'
            AND '2022-05-31'
            AND "vulnerabilities"."present_on_default_branch" = TRUE
            AND "vulnerabilities"."project_id" = "projects"."id")
Before index
Time: 1.400 s
  - planning: 7.874 ms
  - execution: 1.392 s
    - I/O read: 1.367 s
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 3954 (~30.90 MiB) from the buffer pool
  - reads: 1311 (~10.20 MiB) from the OS file cache, including disk I/O
  - dirtied: 40 (~320.00 KiB)
  - writes: 0

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/10506/commands/37650#visualize-depesz

After index
Time: 21.245 ms
  - planning: 9.990 ms
  - execution: 11.255 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 4614 (~36.00 MiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/10506/commands/37654#visualize-depesz

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marcos Rocha

Merge request reports