Get vulnerability_finding from existing queried data
What does this MR do?
Addresses https://gitlab.com/gitlab-org/gitlab/-/issues/338085
- When invoking
Vulnerabilities::CreateServicefromSecurity::StoreReportService,vulnerability_findingis already loaded form DB, but still we pass theidofvulnerability_finding. This triggers an unwanted query inVulnerabilities::CreateServicewhich creates a sub-transaction. This could be avoided by passing thevulnerability_findingrecord toVulnerabilities::CreateService. - The transaction block in
Vulnerabilities::CreateServicerollbacks only whenActiveRecord::RecordNotFoundis raised. This happens when thevulnerability_findingis not found or already has a different vulnerability associated with it. This is a valid case when the service is invoked from the Create Vulnerability API. But when the service is invoked fromSecurity::StoreReportService, we are sure that thevulnerability_findingwould not have a different vulnerability associated with it. This means that we don't need to dolock_for_confirmation!which usesSELECT FOR UPDATEand it is not performant. - Because the transaction fails only for
ActiveRecord::RecordNotFound, we don't need to wrapStatistics::UpdateServiceandHistoricalStatistics::UpdateServicein the transaction.
Improvements
By conditionally removing the sub-transaction in Vulnerabilities::CreateService, we can reduce 1 query everytime the service is invoked, which means if the Security::StoreReportService processes a report with 1000 vulnerabilities, 1000 queries could be reduced from the overall queries count. And since we are moving Statistics::UpdateService and HistoricalStatistics::UpdateService out of the sub-transaction, the number of nested sub-transactions from those 2 services are reduced.
I used Measurable to measure Security::StoreReportsService by running ee/spec/services/security/store_report_service_spec.rb following Stan's comment https://gitlab.com/gitlab-org/gitlab/-/issues/338085#note_648108114 with a test gl-sast-report-json containing 1000 vulnerabilities. These are the results:
Before
I, [2021-08-13T16:58:55.412990 #23242] INFO -- : ---
:class: Security::StoreReportsService
:gc_stats:
:time_to_finish: 44.25615599998855
:number_of_sql_calls: 18713
:memory_usage: 0.0 MiB
:label: process_23242
After
I, [2021-08-13T16:54:18.721069 #22385] INFO -- : ---
:class: Security::StoreReportsService
:gc_stats:
:time_to_finish: 43.77439299999969
:number_of_sql_calls: 17710
:memory_usage: 0.0 MiB
:label: process_22385
Does this MR meet the acceptance criteria?
Conformity
-
I have included changelog trailers, or none are needed. (Does this MR need a changelog?) -
I have added/updated documentation, or it's not needed. (Is documentation required?) -
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?) -
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?) -
I have self-reviewed this MR per code review guidelines. -
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) -
I have followed the style guides. -
This change is backwards compatible across updates, or this does not apply.
Availability and Testing
-
I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.) -
I have tested this MR in all supported browsers, or it's not needed. -
I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.
Security
Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team