Resolve vulnerability_historical_statistics cross join issues
Summary
Vulnerabilities::HistoricalStatistic has scopes that cause cross-database joins b/w
the sec gitlab schema and the main one.
Further details
scope :for_project, ->(project) { where(project: project).allow_cross_joins_across_databases(url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/474140') }
scope :by_projects, ->(values) { where(project_id: values).allow_cross_joins_across_databases(url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/474140') }
Technically these scopes cause cross-database joins when they receive projects, but not when they receive project IDs.
Callers of the by_projects scope
def drop_by_project
MODELS_TO_DROP_BY_PROJECT_ID.each do |model|
loop do
deleted = model.by_projects(project)
.allow_cross_joins_across_databases(url: "https://gitlab.com/groups/gitlab-org/-/epics/14197#cross-db-issues-to-be-resolved")
.limit(BATCH_SIZE)
.delete_all
break if deleted == 0
end
end
end
Quoting @ghavenga's comment:
The current implementation of the removal service only passes a single project as the service deletes one project at a time., making the scope safe to define as above.
Callers of the for_project scope
def vulnerability_historical_statistics
if ::Feature.enabled?(:use_namespace_historical_statistics_for_group_security_dashboard, self)
::Vulnerabilities::NamespaceHistoricalStatistic.for_namespace_and_descendants(self)
else
::Vulnerabilities::HistoricalStatistic.for_project(projects_for_group_and_its_subgroups_without_deleted)
end
end
Quoting @ghavenga's comment:
🔴 This is not simple to fix though. The primary caller of this isResolvers::VulnerabilitiesCountPerDayResolverwhich formulates a complicated query to present in the report as below:
Proposal
One of the cross joins originating from here was dependent on #440712 (closed) which has now been implemented.
As such, we should be able to move forward with fixing this now that the historical statistics are aggregated at the namespace level with the ::Vulnerabilities::NamespaceHistoricalStatistic model.