Resolve cross DB issues in ee/app/models/ee/group.rb

Summary

The group model has various relationships to vulnerability objects that cannot be directly queried anymore. These will need to be remediated for the decomposition.

Further details

Failing specs:

rspec ./ee/spec/models/ee/group_spec.rb:717 # Group#vulnerabilities returns vulnerabilities for all non-deleted projects in the group and its subgroups
rspec ./ee/spec/models/ee/group_spec.rb:764 # Group#vulnerability_scanners returns vulnerability scanners for all non-archived, non-deleted projects in the group and its subgroups
rspec ./ee/spec/models/ee/group_spec.rb:786 # Group#vulnerability_historical_statistics returns vulnerability historical statistics for all non-archived, non-deleted projects in the group and its subgroups

Failure:

     Failure/Error: raise CrossSchemaAccessError, message
     
     Gitlab::Database::QueryAnalyzers::GitlabSchemasValidateConnection::CrossSchemaAccessError:
       The query tried to access ["vulnerabilities", "projects", "namespaces"] (of gitlab_main_cell,gitlab_sec) which is outside of allowed schemas ([:gitlab_internal, :gitlab_sec, :gitlab_shared]) for the current connection 'sec'

Currently #vulnerabilities and #vulnerability_scanners of EE::Group both join sec tables with the projects table using the Project.for_group_and_its_subgroups scope – directly or not.

   def vulnerabilities
      ::Vulnerability.where(
        project: ::Project.for_group_and_its_subgroups(self).without_deleted
      ).allow_cross_joins_across_databases(url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/486216')
    end

    # ...

    def vulnerability_scanners
      ::Vulnerabilities::Scanner.where(project: projects_for_group_and_its_subgroups_without_deleted).allow_cross_joins_across_databases(url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/486216')
    end

    # ...

    def projects_for_group_and_its_subgroups_without_deleted
      ::Project.for_group_and_its_subgroups(self).non_archived.without_deleted
    end

See definition of the scope:

  scope :for_group_and_its_subgroups, ->(group) { where(namespace_id: group.self_and_descendants.select(:id)) }

#vulnerability_historical_statistics causes a similar cross-database join when the feature flag use_namespace_historical_statistics_for_group_security_dashboard is disabled. Flag cleanup issue: [Feature flag] Cleanup `use_namespace_historica... (#507551 - closed) • Gal Katz • 17.9

    def vulnerability_historical_statistics
      if ::Feature.enabled?(:use_namespace_historical_statistics_for_group_security_dashboard, self)
        ::Vulnerabilities::NamespaceHistoricalStatistic.for_namespace_and_descendants(self)
      else
        ::Vulnerabilities::HistoricalStatistic.for_project(projects_for_group_and_its_subgroups_without_deleted)
      end
    end

Ultimately #vulnerability_scanners is used by the ScannersResolver, which is used to render the vulnerability_scanners field for Group resources of the GraphQL API. See https://gitlab.com/gitlab-org/gitlab/-/blob/75d49fe13646e1e0d3b68233ac4a965c86853917/ee/app/graphql/ee/types/group_type.rb#L72

        field :vulnerability_scanners, ::Types::VulnerabilityScannerType.connection_type,
          null: true,
          description: 'Vulnerability scanners reported on the project vulnerabilities of the group and ' \
                       'its subgroups.',
          resolver: ::Resolvers::Vulnerabilities::ScannersResolver

#vulnerabilities is used by the Security::VulnerabilitiesFinder but this finder is being removed as part of Replace VulnerabilitiesFinder with Vulnerabilit... (!175947 - merged).

#vulnerabilities is also used EE::Banzai::Filter::References#parent_records, and this needs to be addressed.

Proposal

  • Join w/ vulnerability_reads or vulnerability_statistics. Leverage traversal_ids to select records in a group, and to collect project IDs.
  • Link to feature flag cleanup issue if the code behind a feature flag solves the cross-join.

Implementation plan

Edited by Fabien Catteau