Resolve cross-join in ProjectsGrade.grades_for with groups
What does this MR do and why?
Remove cross-database join from Vulnerabilities::ProjectsGrade.grades_for
when vulnerable objects are Group models.
This is under a feature flag and can only be tested in the context of groups, not instances.
-
Introduce new class method
.grades_for_vulnerables_no_cross_join.This method behaves like
.grades_forbut doesn't have a cross-database join. It iterates batches of sub-groups and sub-batches of projects IDs to collectvulnerability_statistics. -
Introduce
remove_cross_join_from_vulnerabilities_projects_gradeflag..grades_fordelegates to.grades_for_vulnerables_no_cross_joinwhen all the vulnerable objects it receives are groups where the flag is enabled. It doesn't check whether the feature flag is enabled for sub-groups. -
Add a spec for when
.grades_forreceives multiple groups.This spec ensures that the behavior doesn't change even though this is not the desired behavior. See ProjectsGrade.grades_for returns same grades fo... (#507992 - closed).
-
Update two unrelated specs to make Rubocop happy.
Please note that .grades_for might also receive an InstanceSecurityDashboard
but this MR doesn't handle that case, for the following reasons:
- This would require a separate feature flag since the actor is different.
- We'd better iterate first, and check how the new implementation performs for large groups.
- We should probably have a distinct logic for
InstanceSecurityDashboardanyway, and leverage its#projectsmethod. Then we would have to address Resolve cross DB issues in ee/app/models/instan... (#485658 - closed). See #485658 (comment 2252636338)
Next steps
- Enable the feature flag for groups we own on gitlab.com (like
gitlab-org), and make sure that new implementation (that removes cross-database joins) performs well. - If it does, then roll out and remove the feature flag.
- Implement a similar change for the
InstanceSecurityDashboard.
References
Please include cross links to any resources that are relevant to this MR This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
bundle exec rspec ee/spec/models/vulnerabilities/projects_grade_spec.rb
- When
secis shared, all specs pass. - When
secis separate, specs for.grades_forfail for instances, or for groups when the flag is disabled.
rspec ./ee/spec/models/vulnerabilities/projects_grade_spec.rb:60 # Vulnerabilities::ProjectsGrade.grades_for when the given vulnerable is a Group when subgroups are not included when the filter is not given when remove_cross_join_from_vulnerabilities_projects_grade is disabled returns the letter grades for given vulnerable
rspec ./ee/spec/models/vulnerabilities/projects_grade_spec.rb:85 # Vulnerabilities::ProjectsGrade.grades_for when the given vulnerable is a Group when subgroups are not included when the filter is given when remove_cross_join_from_vulnerabilities_projects_grade is disabled returns the filtered letter grade for given vulnerable
rspec ./ee/spec/models/vulnerabilities/projects_grade_spec.rb:117 # Vulnerabilities::ProjectsGrade.grades_for when the given vulnerable is a Group when subgroups are included when the filter is not given when remove_cross_join_from_vulnerabilities_projects_grade is disabled returns the letter grades for given vulnerable
rspec ./ee/spec/models/vulnerabilities/projects_grade_spec.rb:136 # Vulnerabilities::ProjectsGrade.grades_for when the given vulnerable is a Group when subgroups are included when the filter is not given when multiple batches are required when remove_cross_join_from_vulnerabilities_projects_grade is disabled returns the letter grades for given vulnerable
rspec ./ee/spec/models/vulnerabilities/projects_grade_spec.rb:162 # Vulnerabilities::ProjectsGrade.grades_for when the given vulnerable is a Group when subgroups are included when the filter is given when remove_cross_join_from_vulnerabilities_projects_grade is disabled returns the filtered letter grade for given vulnerable
rspec ./ee/spec/models/vulnerabilities/projects_grade_spec.rb:196 # Vulnerabilities::ProjectsGrade.grades_for when the given vulnerables are groups when remove_cross_join_from_vulnerabilities_projects_grade is disabled in one of the groups returns all letter grades for each vulnerable
rspec ./ee/spec/models/vulnerabilities/projects_grade_spec.rb:227 # Vulnerabilities::ProjectsGrade.grades_for when the given vulnerable is an InstanceSecurityDashboard when the filter is not given returns the letter grades for given vulnerable
rspec ./ee/spec/models/vulnerabilities/projects_grade_spec.rb:242 # Vulnerabilities::ProjectsGrade.grades_for when the given vulnerable is an InstanceSecurityDashboard when the filter is given returns the filtered letter grade for given vulnerable
Related to #503387 (closed)