Skip to content

Get vulnerability_finding from existing queried data

Sashi Kumar Kumaresan requested to merge sk/338085-remove-transaction into master

What does this MR do?

Addresses https://gitlab.com/gitlab-org/gitlab/-/issues/338085

  1. When invoking Vulnerabilities::CreateService from Security::StoreReportService, vulnerability_finding is already loaded form DB, but still we pass the id of vulnerability_finding. This triggers an unwanted query in Vulnerabilities::CreateService which creates a sub-transaction. This could be avoided by passing the vulnerability_finding record to Vulnerabilities::CreateService.
  2. The transaction block in Vulnerabilities::CreateService rollbacks only when ActiveRecord::RecordNotFound is raised. This happens when the vulnerability_finding is not found or already has a different vulnerability associated with it. This is a valid case when the service is invoked from the Create Vulnerability API. But when the service is invoked from Security::StoreReportService, we are sure that the vulnerability_finding would not have a different vulnerability associated with it. This means that we don't need to do lock_for_confirmation! which uses SELECT FOR UPDATE and it is not performant.
  3. Because the transaction fails only for ActiveRecord::RecordNotFound, we don't need to wrap Statistics::UpdateService and HistoricalStatistics::UpdateService in the transaction.

Improvements

By conditionally removing the sub-transaction in Vulnerabilities::CreateService, we can reduce 1 query everytime the service is invoked, which means if the Security::StoreReportService processes a report with 1000 vulnerabilities, 1000 queries could be reduced from the overall queries count. And since we are moving Statistics::UpdateService and HistoricalStatistics::UpdateService out of the sub-transaction, the number of nested sub-transactions from those 2 services are reduced.

I used Measurable to measure Security::StoreReportsService by running ee/spec/services/security/store_report_service_spec.rb following Stan's comment https://gitlab.com/gitlab-org/gitlab/-/issues/338085#note_648108114 with a test gl-sast-report-json containing 1000 vulnerabilities. These are the results:

gl-sast-report.json

Before

I, [2021-08-13T16:58:55.412990 #23242]  INFO -- : ---
:class: Security::StoreReportsService
:gc_stats: 
:time_to_finish: 44.25615599998855
:number_of_sql_calls: 18713
:memory_usage: 0.0 MiB
:label: process_23242

After

I, [2021-08-13T16:54:18.721069 #22385]  INFO -- : ---
:class: Security::StoreReportsService
:gc_stats: 
:time_to_finish: 43.77439299999969
:number_of_sql_calls: 17710
:memory_usage: 0.0 MiB
:label: process_22385

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Sashi Kumar Kumaresan

Merge request reports