Load artifacts based on the finding metadata
What does this MR do?
This MR introduces a new way to load report artifacts to build data for the "pipeline security tab". Before this MR, we were trying to load all the artifacts into memory which was causing timeout errors for the pipelines with lots of security scans. With this MR, we are trying to load only the necessary artifacts into memory for the given parameters(filters and pagination) therefore this will fix the timeout issues.
Here is a diagram to describe the overall idea;
Related to #267021 (closed).
Database migrations
This MR introduces a new index for vulnerability_occurrences
table on project_fingerprint
column.
rake db:migrate:up
== 20201110035029 CreatedIndexForVulnerabilityOccurrencesOnProjectFingerprint: migrating
-- transaction_open?()
-> 0.0000s
-- index_exists?(:vulnerability_occurrences, :project_fingerprint, {:name=>"index_vulnerability_occurrences_on_project_fingerprint", :algorithm=>:concurrently})
-> 0.0043s
-- add_index(:vulnerability_occurrences, :project_fingerprint, {:name=>"index_vulnerability_occurrences_on_project_fingerprint", :algorithm=>:concurrently})
-> 0.0046s
== 20201110035029 CreatedIndexForVulnerabilityOccurrencesOnProjectFingerprint: migrated (0.0094s)
rake db:migrate:down
== 20201110035029 CreatedIndexForVulnerabilityOccurrencesOnProjectFingerprint: reverting
-- transaction_open?()
-> 0.0000s
-- indexes(:vulnerability_occurrences)
-> 0.0039s
-- remove_index(:vulnerability_occurrences, {:algorithm=>:concurrently, :name=>"index_vulnerability_occurrences_on_project_fingerprint"})
-> 0.0022s
== 20201110035029 CreatedIndexForVulnerabilityOccurrencesOnProjectFingerprint: reverted (0.0065s)
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry - [-] Documentation (if required)
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
- [-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
- [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team