Skip to content

Uses eager loading to avoid N+1 queries

James Johnson requested to merge pipeline_vulnerabilities_finder_n_1_fix into master

What does this MR do?

This merge request uses eager loading to eagerly load the referenced vulnerability models of vulnerability findings.

The performance problems involved with the pipeline_vulnerabilities_finder were reported in this issue: #321981 (closed)

Screenshots (strongly suggested)

N/A

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Database

Migration to Add Index

UP:

$> rake db:migrate:up VERSION=20210310000627
== 20210310000627 AddIdxVulnerabilityOccurrencesDedup: migrating ==============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:vulnerability_occurrences, [:project_id, :report_type, :project_fingerprint], {:name=>"index_vulnerability_occurrences_deduplication", :algorithm=>:concurrently})
   -> 0.0039s
-- execute("SET statement_timeout TO 0")
   -> 0.0004s
-- add_index(:vulnerability_occurrences, [:project_id, :report_type, :project_fingerprint], {:name=>"index_vulnerability_occurrences_deduplication", :algorithm=>:concurrently})
   -> 0.0125s
-- execute("RESET ALL")
   -> 0.0010s
== 20210310000627 AddIdxVulnerabilityOccurrencesDedup: migrated (0.0190s) =====

DOWN:

$> rake db:migrate:down VERSION=20210310000627
== 20210310000627 AddIdxVulnerabilityOccurrencesDedup: reverting ==============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:vulnerability_occurrences, [:project_id, :report_type, :project_fingerprint], {:name=>"index_vulnerability_occurrences_deduplication", :algorithm=>:concurrently})
   -> 0.0044s
-- execute("SET statement_timeout TO 0")
   -> 0.0004s
-- remove_index(:vulnerability_occurrences, {:name=>"index_vulnerability_occurrences_deduplication", :algorithm=>:concurrently, :column=>[:project_id, :report_type, :project_fingerprint]})
   -> 0.0132s
-- execute("RESET ALL")
   -> 0.0006s
== 20210310000627 AddIdxVulnerabilityOccurrencesDedup: reverted (0.0193s) =====

postgres.ai

https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/2571/commands/7921

Modified Query Raw SQL

Below is an equivalent raw SQL that would run if the report_type was sast (0), and the project was gitlab-org/gitlab (278964):

Raw SQL (expand me)
SELECT
    "vulnerability_occurrences"."vulnerability_id", "vulnerability_occurrences"."project_fingerprint"
FROM
    "vulnerability_occurrences"
WHERE
    "vulnerability_occurrences"."project_id" = 278964
    AND "vulnerability_occurrences"."report_type" = 0
    AND "vulnerability_occurrences"."project_fingerprint" IN ('\x2cb10cd78b6a5157cd72af9d3acbac87481dded1', '\x37ed4483fa5d91529cfaf95b071f623c99e8b766', '\x7d6220e5971a5b8e8deb554e07b62419331c2c96', '\xfa718ba408acaf5575f9d663625d6831a933df39', '\xf46ddd38779f0f36339e9fb08ef0154b840c7eb0', '\x3af27ef92610e350f6938b4fc591cb13e6c761fe', '\x3abc4170baa09bc6316b08825488b52e1e4e4e53', '\xc90acfdfb55a2cf3d9e555bca3a4a27d893cad79', '\x13ccdfc084779097ed9e062a5196c8d12fb763cd', '\x75347c87eb417ea074a99b58de0465c0a33c4ed1', '\x086fe7cde18b0951bd508d9220316f69dbd71b88', '\xb272e22e16207c86efa5fe714f23cca4686aec02', '\xd6724a60898e5287bd3b8c7b37e7c52d74fe666b', '\x493b6ea68051223bc5caf8cdcf2735b5ba18ed79', '\x28e7211e0c7653f6a2d9dffa35002a7e8f7240c8', '\x712913b85d5e75708f0a45d23f874b350f2d4592', '\xf4257527591d5c9846b0ded602ca696abe7b3ec6', '\x95ee3e935b522303f2b21505c0bd5b16f53c4f56', '\x4ce5a6d7fd75a0d837dc694fff341b0070adb5aa', '\x07d4dea45d5722b53df29ae2f6b340128e4a86fd', '\x8c9d819d470fd9eee295816e5bab226b0b6314d8', '\xc70638d48298e62cd1fc6c3a9ff5ed91761520e7', '\x9469761b984bbc39a7b5cd7eeef6c999a6856e74', '\x12464f804d5feafe40a7213dd377ec37be897ec8', '\xb0208f558d67806a5927a396d5d0cf50b9b40b14', '\x1276882c454e107bcd7073e56b3e00662e24f9a2', '\xfd8a61f3804c71be8c9e9b43a8dd2368d4e784c5', '\xd2d9527b8ad358e642ce8bb80168189ac4c0f92d', '\xb8cac2cbf4a9934a5cfcdcd6d6d9fcb278bb2e0c', '\x3b2b7c5f02ee8fac9f755f4864a04c119d547037', '\x4b214a9afa81596aad24c44c881e8861a2fb422c', '\xcd784714e656d0508eb5de83c57852abba443b10', '\x790f030f1e0ada3eb66297fef9c2f2013e8429cc', '\xfbe5ed703b49aa4a195bfdfb044a1487bfd1edd4', '\x6f169adb9fdc6e070288c758735197facb5743f6', '\x0c5a02cad1d09da212f4a511574e333ac247e5d7', '\x330ee1015840bdf3e71b10d151f3f8bde0f2e1f7', '\x7d38013a863be1dbb27fba193c28ed364e7f23a7', '\xa0bb511e8a5c149afffab92ea5c684a1f90c897a', '\x4f57caba3bdef9f1cf46583a886e7ee7efbed616', '\x9b21292e84fad2bcb892affa42dc23c52bc03380', '\xe172ddfd575428e1d262cd188e700a5bafde75c8', '\x0ae2f33373465cdc40cf7810970ce419c0983476', '\x4d9f2d244c32a03941731a806555f28ad688f365', '\x34d3a7990b19f10a2e88761c77eaed0607130d52', '\x47c914c9d598b0fbbdf74c43af9b16ef52de3097', '\x6e7870b4f9b019db8da4ef3e456dfde10ec7c4b3', '\x7753b8efa2f84c3659f8769190c8b5294db1336e', '\x8bd46f29e3e4ba2d3c923d912e0ba125111b56df', '\xc7c2321af872588408d6b83369276966c231e04d', '\xc237d82109ed8ddf26bb31de4e9c6a17e39c28cd', '\xb1460a13ceb8ab66aa9fb8ad8fc9266706d3082a', '\xe7d3e1dbc21e236cac53bb62080930394137f757', '\x1c91d29464743cb462da9ba64f45347f95996618', '\x675701bd821cc8974c6a052bde922ab0c9d12823', '\xb99836e4957d693ffd67544b7aa1ef95d545516b', '\x1c06f3064566090425b614b33169e0524c2d5b4d', '\xf07ef1447c974cf0e8c0fa105f2028bff8cfce9f', '\x87577d3cc41538f3806720f67e8914f29266fd93', '\x67949971a96cb81f140d98b0e09386e23684cde1', '\xde05feed890a1173aa69f6cc706b00c23cb3d32d', '\x33baf866af43dcf2d4bd043bdb5f0ca71656847a', '\xe2bd2094476cf949d4850b58106cd39acde33bcf', '\xddab6e825299e820c3132149c32d1e0decbc46e0', '\x5b44e2b22e07525649cebc631496b62e27347fdb', '\x3e88fea4324982dec6f5693571a01c7c3eeff727', '\x7a1a1ab8634f69ee8a1f14db03a97bfe4e978d22')

Info From #database-lab

Summary

Summary:
Time: 137.113 ms
  - planning: 3.597 ms
  - execution: 133.516 ms
    - I/O read: 127.333 ms
    - I/O write: N/A

Shared buffers:
  - hits: 470 (~3.70 MiB) from the buffer pool
  - reads: 44 (~352.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0
Database Plan (expand me)
Bitmap Heap Scan on public.vulnerability_occurrences  (cost=1890.57..1932.43 rows=26 width=29) (actual time=52.056..133.298 rows=114 loops=1)
   Buffers: shared hit=470 read=44
   I/O Timings: read=127.333
   ->  BitmapAnd  (cost=1890.57..1890.57 rows=26 width=0) (actual time=52.002..52.003 rows=0 loops=1)
         Buffers: shared hit=381 read=22
         I/O Timings: read=46.779
         ->  Bitmap Index Scan using index_vulnerability_occurrences_on_project_fingerprint  (cost=0.00..160.47 rows=3048 width=0) (actual time=49.577..49.577 rows=4511 loops=1)
               Index Cond: (vulnerability_occurrences.project_fingerprint = ANY ('{"\\x2cb10cd78b6a5157cd72af9d3acbac87481dded1","\\x37ed4483fa5d91529cfaf95b071f623c99e8b766","\\x7d6220e5971a5b8e8deb554e07b62419331c2c96","\\xfa718ba408acaf5575f9d663625d6831a933df39","\\xf46ddd38779f0f36339e9fb08ef0154b840c7eb0","\\x3af27ef92610e350f6938b4fc591cb13e6c761fe","\\x3abc4170baa09bc6316b08825488b52e1e4e4e53","\\xc90acfdfb55a2cf3d9e555bca3a4a27d893cad79","\\x13ccdfc084779097ed9e062a5196c8d12fb763cd","\\x75347c87eb417ea074a99b58de0465c0a33c4ed1","\\x086fe7cde18b0951bd508d9220316f69dbd71b88","\\xb272e22e16207c86efa5fe714f23cca4686aec02","\\xd6724a60898e5287bd3b8c7b37e7c52d74fe666b","\\x493b6ea68051223bc5caf8cdcf2735b5ba18ed79","\\x28e7211e0c7653f6a2d9dffa35002a7e8f7240c8","\\x712913b85d5e75708f0a45d23f874b350f2d4592","\\xf4257527591d5c9846b0ded602ca696abe7b3ec6","\\x95ee3e935b522303f2b21505c0bd5b16f53c4f56","\\x4ce5a6d7fd75a0d837dc694fff341b0070adb5aa","\\x07d4dea45d5722b53df29ae2f6b340128e4a86fd","\\x8c9d819d470fd9eee295816e5bab226b0b6314d8","\\xc70638d48298e62cd1fc6c3a9ff5ed91761520e7","\\x9469761b984bbc39a7b5cd7eeef6c999a6856e74","\\x12464f804d5feafe40a7213dd377ec37be897ec8","\\xb0208f558d67806a5927a396d5d0cf50b9b40b14","\\x1276882c454e107bcd7073e56b3e00662e24f9a2","\\xfd8a61f3804c71be8c9e9b43a8dd2368d4e784c5","\\xd2d9527b8ad358e642ce8bb80168189ac4c0f92d","\\xb8cac2cbf4a9934a5cfcdcd6d6d9fcb278bb2e0c","\\x3b2b7c5f02ee8fac9f755f4864a04c119d547037","\\x4b214a9afa81596aad24c44c881e8861a2fb422c","\\xcd784714e656d0508eb5de83c57852abba443b10","\\x790f030f1e0ada3eb66297fef9c2f2013e8429cc","\\xfbe5ed703b49aa4a195bfdfb044a1487bfd1edd4","\\x6f169adb9fdc6e070288c758735197facb5743f6","\\x0c5a02cad1d09da212f4a511574e333ac247e5d7","\\x330ee1015840bdf3e71b10d151f3f8bde0f2e1f7","\\x7d38013a863be1dbb27fba193c28ed364e7f23a7","\\xa0bb511e8a5c149afffab92ea5c684a1f90c897a","\\x4f57caba3bdef9f1cf46583a886e7ee7efbed616","\\x9b21292e84fad2bcb892affa42dc23c52bc03380","\\xe172ddfd575428e1d262cd188e700a5bafde75c8","\\x0ae2f33373465cdc40cf7810970ce419c0983476","\\x4d9f2d244c32a03941731a806555f28ad688f365","\\x34d3a7990b19f10a2e88761c77eaed0607130d52","\\x47c914c9d598b0fbbdf74c43af9b16ef52de3097","\\x6e7870b4f9b019db8da4ef3e456dfde10ec7c4b3","\\x7753b8efa2f84c3659f8769190c8b5294db1336e","\\x8bd46f29e3e4ba2d3c923d912e0ba125111b56df","\\xc7c2321af872588408d6b83369276966c231e04d","\\xc237d82109ed8ddf26bb31de4e9c6a17e39c28cd","\\xb1460a13ceb8ab66aa9fb8ad8fc9266706d3082a","\\xe7d3e1dbc21e236cac53bb62080930394137f757","\\x1c91d29464743cb462da9ba64f45347f95996618","\\x675701bd821cc8974c6a052bde922ab0c9d12823","\\xb99836e4957d693ffd67544b7aa1ef95d545516b","\\x1c06f3064566090425b614b33169e0524c2d5b4d","\\xf07ef1447c974cf0e8c0fa105f2028bff8cfce9f","\\x87577d3cc41538f3806720f67e8914f29266fd93","\\x67949971a96cb81f140d98b0e09386e23684cde1","\\xde05feed890a1173aa69f6cc706b00c23cb3d32d","\\x33baf866af43dcf2d4bd043bdb5f0ca71656847a","\\xe2bd2094476cf949d4850b58106cd39acde33bcf","\\xddab6e825299e820c3132149c32d1e0decbc46e0","\\x5b44e2b22e07525649cebc631496b62e27347fdb","\\x3e88fea4324982dec6f5693571a01c7c3eeff727","\\x7a1a1ab8634f69ee8a1f14db03a97bfe4e978d22"}'::bytea[]))
               Buffers: shared hit=264 read=22
               I/O Timings: read=46.779
         ->  Bitmap Index Scan using tmp_idx_deduplicate_vulnerability_occurrences  (cost=0.00..1729.84 rows=66578 width=0) (actual time=2.096..2.096 rows=6942 loops=1)
               Index Cond: ((vulnerability_occurrences.project_id = 278964) AND (vulnerability_occurrences.report_type = 0))
               Buffers: shared hit=117
Edited by James Johnson

Merge request reports