Make security related project archival features compatible with group archival

Prerequisite

Before you start, please check out the parent epic description to get the full context: Make project archival features compatible with ... (&19690)

Description

We ran a search for "archive" and "archival" in the codebase and pulled together the following relevant results for the security domain. We'd need your help with:

1. Review the search results and update the code to use our new ancestor-aware methods and scopes wherever needed
2. Creating new services/workers wherever they're needed
3. Note: Not all "archive" mentions are about project archival – feel free to skip anything that's not relevant!

Archive/Archival Occurrences

Security / Vulnerabilities

1. ee/app/models/security/project_statistics.rb

   • project_ids = ::Vulnerabilities::Statistic.by_group(group).unarchived.select(:project_id)

2. ee/app/models/ee/project.rb

   • has_many :vulnerability_archives, class_name: 'Vulnerabilities::Archive'

3. ee/app/models/ee/group.rb

   • ::Vulnerability.where(id: Vulnerabilities::Read.by_group(self).select(:vulnerability_id).unarchived)
   • ::Vulnerabilities::Scanner.where(project: Vulnerabilities::Statistic.by_group(self).unarchived.select(:project_id))

4. ee/app/models/vulnerabilities/statistic.rb

   • scope :unarchived, -> { where(archived: false) }
   • INSERT INTO %<table_name>s AS target (project_id, archived, traversal_ids, latest_pipeline_id, letter_grade, created_at, updated_at)
   • VALUES (%{project_id}, %{archived}, %{traversal_ids}, %<latest_pipeline_id>d, %<letter_grade>d, now(), now())
   • archived: project.archived,
   • INSERT INTO %<table_name>s AS target (project_id, archived, traversal_ids, latest_pipeline_id, letter_grade, created_at, updated_at)
   • pipeline.project.archived,

5. ee/app/models/vulnerabilities/projects_grade.rb

   • .unarchived_projects_for(::Project.id_in(project_ids))
   • unarchived_projects_for(projects)
   • def self.unarchived_projects_for(projects)
   • projects.non_archived
   • .unarchived

6. ee/app/models/vulnerabilities/read.rb

   • scope :unarchived, -> { where(archived: false) }
   • by_group(group).unarchived.loose_index_scan(column: :traversal_ids)

7. ee/app/models/vulnerabilities/identifier.rb

   • project_ids = ::Vulnerabilities::Statistic.by_group(group).unarchived.select(:project_id)

8. ee/app/finders/security/vulnerability_elastic_base_finder.rb

   • # include_archived_projects: defaulted to false. Determines if results will include vulnerabilities
   • # associated with archived projects
   • include_archived_projects: params[:include_archived_projects],

9. ee/app/finders/security/security_policy_projects_finder.rb

   • .non_archived

10. ee/app/finders/security/vulnerability_reads_finder.rb

    • # include_archived_projects: defaulted to false. Determines if results will include vulnerabilities
    • # associated with archived projects
    • filter_archived_projects
    • def filter_archived_projects
    • return if params[:include_archived_projects] == true
    • @vulnerability_reads = vulnerability_reads.unarchived

11. ee/app/workers/vulnerabilities/update_archived_attribute_of_vulnerability_reads_worker.rb

    • class UpdateArchivedAttributeOfVulnerabilityReadsWorker
    • Vulnerabilities::UpdateArchivedOfVulnerabilityReadsService.execute(project_id)

12. ee/app/workers/vulnerabilities/update_archived_attribute_of_vulnerability_statistics_worker.rb

    • class UpdateArchivedAttributeOfVulnerabilityStatisticsWorker
    • Vulnerabilities::UpdateArchivedOfVulnerabilityStatisticsService.execute(project_id)

13. ee/app/workers/vulnerabilities/process_archived_events_worker.rb

    • # Ingest archived events to enqueue updating of denormalized column.
    • class ProcessArchivedEventsWorker
    • Vulnerabilities::UpdateArchivedOfVulnerabilityReadsService.execute(project_id)
    • Vulnerabilities::UpdateArchivedOfVulnerabilityStatisticsService.execute(project_id)

14. ee/app/workers/vulnerabilities/namespace_statistics/process_group_delete_events_worker.rb

    • Statistic.unarchived.within(traversal_ids)

15. ee/app/services/vulnerabilities/archival/restoration/tasks/adjust_traversal_ids_and_archived_attributes.rb

    • # Same applies to the archived attribute which is also handled by this
    • class AdjustTraversalIdsAndArchivedAttributes
    • archived = map.archived
    • (%{values}) AS map(vulnerability_id, traversal_ids, archived)
    • vulnerability_reads.archived != map.archived
    • project.archived
    • # archived attributes of vulnerability_reads records by loading the related projects into memory.
    • # To address this, we check if the traversal_ids or archived attributes are changed after we update
    • ensure_archived_consistency
    • def ensure_archived_consistency
    • projects_which_archived_changed_after_update.each do |changed_project|
    • Vulnerabilities::ProcessArchivedEventsWorker.perform_async(
    • 'Projects::ProjectArchivedEvent',
    • def projects_which_archived_changed_after_update
    • project_before_update.archived != project_after_update.archived

16. ee/app/services/vulnerability_exports/export_service.rb

    • exportable.vulnerability_reads.unarchived.order_traversal_ids_asc

17. ee/app/services/security/vulnerability_scanning/create_vulnerability_service.rb

    • schedule_updating_archived_status_if_needed
    • def schedule_updating_archived_status_if_needed
    • return unless map_of_projects_to_adjust[:archival_updated].present?
    • Vulnerabilities::UpdateArchivedAttributeOfVulnerabilityReadsWorker.bulk_perform_async_with_contexts(
    • map_of_projects_to_adjust[:archival_updated],
    • add_into_archival_updated_list_if_needed(map, project, project_with_most_recent_changes)
    • def add_into_archival_updated_list_if_needed(map, project, project_with_most_recent_changes)
    • return if project.archived == project_with_most_recent_changes.archived
    • map[:archival_updated] << project

18. ee/app/services/vulnerabilities/starboard_vulnerability_create_service.rb

    • process_archival_and_traversal_ids_changes if vulnerability_service_response&.success?

19. ee/app/services/security/analyzers_status/update_archived_service.rb

    • class UpdateArchivedService
    • project.analyzer_statuses.update!(archived: project.archived)

20. ee/app/services/security/ingestion/tasks/ingest_vulnerability_statistics.rb

    • AS target (project_id, archived, traversal_ids, letter_grade, created_at, updated_at, %{insert_attributes})
    • project.archived,

21. ee/app/services/security/ingestion/tasks/ingest_vulnerability_reads/upsert.rb

    • archived: vulnerability.project.archived,

22. ee/app/services/security/ingestion/schedule_mark_dropped_as_resolved_service.rb

    • .by_projects(::Project.non_archived.id_in(project_id).pluck_primary_key)

23. ee/app/services/security/ingestion/ingest_reports_service.rb

    • @original_archived_value = project.archived
    • schedule_updating_archived_status_if_needed
    • attr_reader :pipeline, :original_archived_value, :original_traversal_ids_value
    • def schedule_updating_archived_status_if_needed
    • return unless archived_value_changed?
    • Vulnerabilities::UpdateArchivedAttributeOfVulnerabilityReadsWorker.perform_async(project.id)
    • def archived_value_changed?
    • reloaded_project.archived != original_archived_value

24. ee/app/services/security/ingestion/tasks/ingest_vulnerability_reads/update.rb

    • archived: finding_map.project.archived,

25. ee/app/services/security/inventory_filters/vulnerability_statistics_update_service.rb

    • AS target (project_id, project_name, archived, traversal_ids, %{insert_attributes})
    • archived = EXCLUDED.archived,
    • project.archived,

26. ee/app/services/security/inventory_filters/analyzer_status_update_service.rb

    • archived: project.archived

27. ee/app/services/security/analyzer_namespace_statuses/adjustment_service.rb

    • ON analyzer_project_statuses.archived = FALSE
    • -- Handle case where all projects are archived: return 0 counts for existing analyzer types
    • WHERE analyzer_project_statuses.archived = FALSE

28. ee/app/services/security/analyzer_namespace_statuses/find_namespaces_with_analyzer_statuses_service.rb

    • analyzer_project_statuses.archived = false

29. ee/app/services/vulnerabilities/statistics/update_service.rb

    • INSERT INTO #{Statistic.table_name} AS target (project_id, archived, traversal_ids, %{insert_attributes}, letter_grade, created_at, updated_at)
    • VALUES (%{project_id}, %{archived}, %{traversal_ids}, %{insert_values}, %{letter_grade}, now(), now())
    • archived: project.archived,

30. ee/app/services/vulnerabilities/statistics/adjustment_service.rb

    • (project_id, archived, traversal_ids, total, info, unknown, low, medium, high, critical, letter_grade, created_at, updated_at)
    • project_attributes.archived AS archived,
    • JOIN (%{project_attributes}) project_attributes(project_id, archived, traversal_ids)
    • SELECT project_id, archived, traversal_ids, total, info, unknown, low, medium, high, critical, letter_grade, created_at, updated_at
    • .pluck(:id, :archived, :traversal_ids, :namespace_id)

31. ee/app/services/vulnerabilities/create_service_base.rb

    • @original_archived_value = project.archived
    • attr_reader :author, :project, :original_archived_value, :original_traversal_ids_value
    • def process_archival_and_traversal_ids_changes
    • schedule_updating_archived_status_if_needed
    • def schedule_updating_archived_status_if_needed
    • return if original_archived_value == reloaded_project.archived
    • Vulnerabilities::UpdateArchivedAttributeOfVulnerabilityReadsWorker.perform_async(project.id)

32. ee/app/services/vulnerabilities/manually_create_service.rb

    • process_archival_and_traversal_ids_changes if response.success?

33. ee/app/services/vulnerabilities/reads/upsert_service.rb

    • archived: vulnerability.project.archived?,

34. ee/app/services/vulnerabilities/namespace_statistics/find_vulnerable_namespaces_service.rb

    • vulnerability_statistics.archived = false

35. ee/app/services/vulnerabilities/namespace_statistics/adjustment_service.rb

    • ON vulnerability_statistics.archived = FALSE

36. ee/lib/quality/seeders/vulnerabilities.rb

    • archived: vulnerability.project.archived,

37. ee/lib/search/elastic/vulnerability_query_builder.rb

    • query_hash = ::Search::Elastic::VulnerabilityFilters.by_archived_projects(

38. ee/lib/search/elastic/types/vulnerability.rb

    • archived: { type: 'boolean' },

39. ee/lib/search/elastic/vulnerability_filters.rb

    • def by_archived_projects(query_hash:, options:)
    • include_archived_projects = !!options[:include_archived_projects]
    • return query_hash if include_archived_projects
    • { term: { archived: { _name: context.name(:archived_projects), value: false } } }

40. app/presenters/projects/security/configuration_presenter.rb

    • !archived?

41. ee/app/models/security/analyzer_namespace_status.rb

    • @total_projects_count ||= group.all_unarchived_project_ids.size

42. ee/app/models/security/analyzer_project_status.rb

    • scope :unarchived, -> { where(archived: false) }

43. ee/app/models/security/inventory_filter.rb

    • validates :archived, allow_nil: false, inclusion: { in: [true, false] }

44. ee/app/models/security/project_tracked_context.rb

    • event :archive do

45. ee/app/graphql/resolvers/security/namespace_security_projects_resolver.rb

    • finder_params = { include_subgroups: args[:include_subgroups], include_archived: false, ids: ids }

46. ee/app/workers/security/analyzers_status/process_archived_events_worker.rb

    • class ProcessArchivedEventsWorker
    • UpdateArchivedService.execute(project)
    • Security::InventoryFilter.by_project_id(project.id).update(archived: project.archived)

47. ee/app/workers/security/analyzer_namespace_statuses/process_group_deleted_events_worker.rb

    • AnalyzerProjectStatus.unarchived.within(traversal_ids)

48. ee/app/services/concerns/security/analyzers_status/aggregated_types_handler.rb

    • archived: project.archived,

49. ee/app/services/concerns/security/scan_result_policies/policy_violation_comment_generator.rb

    • return true if project.archived?

50. ee/app/models/instance_security_dashboard.rb

    • def non_archived_project_ids
    • projects.non_archived.limit(limit).pluck_primary_key
    • project_ids = non_archived_project_ids
    • project_ids = non_archived_project_ids
    • project_ids = non_archived_project_ids

51. ee/lib/ee/gitlab/background_migration/backfill_security_inventory_filters.rb

    • statistics_records = sub_batch.select(:project_id, :traversal_ids, :archived, *SEVERITY_COLUMNS)
    • archived: stat.archived,

52. lib/gitlab/background_migration/backfill_archived_and_traversal_ids_to_vulnerability_reads.rb

    • class BackfillArchivedAndTraversalIdsToVulnerabilityReads < BatchedMigrationJob
    • operation_name :backfill_archived_and_traversal_ids_in_vulnerability_reads_table
    • archived = projects.archived

53. lib/gitlab/background_migration/backfill_archived_and_traversal_ids_to_vulnerability_statistics.rb

    • class BackfillArchivedAndTraversalIdsToVulnerabilityStatistics < BatchedMigrationJob
    • operation_name :backfill_archived_and_traversal_ids_in_vulnerability_statistics_table
    • WITH project_to_archived_traversal_ids (project_id, archived, traversal_ids) AS (
    • traversal_ids = project_to_archived_traversal_ids.traversal_ids,
    • archived = project_to_archived_traversal_ids.archived
    • project_to_archived_traversal_ids
    • project_to_archived_traversal_ids.project_id = vulnerability_statistics.project_id AND
    • scope :archived_and_traversal_ids, ->(project_ids) {
    • .pluck(:id, :archived, :traversal_ids)
    • project_info = Project.archived_and_traversal_ids(project_ids)

54. ee/lib/ee/gitlab/background_migration/backfill_analyzer_project_statuses.rb

    • projects_info = sub_batch.pluck(:project_id, :traversal_ids, :latest_pipeline_id, :archived)
    • project_info.each do |project_id, traversal_ids, pipeline_id, archived|
    • statuses = process_builds(builds, project_id, traversal_ids, archived)
    • def process_builds(builds, project_id, traversal_ids, archived)
    • archived: archived
    • %w[project_id traversal_ids analyzer_type status last_call build_id archived created_at updated_at]
    • status_data[:archived],

SBOM

1. ee/app/models/ee/group.rb

   • Sbom::Occurrence.for_namespace_and_descendants(self).unarchived

2. ee/app/services/sbom/ingestion/tasks/ingest_occurrences.rb

   • archived: project.archived,

3. ee/app/services/sbom/sync_archived_status_service.rb

   • class SyncArchivedStatusService
   • in_lock(lease_key, ttl: LEASE_TTL, sleep_sec: LEASE_TRY_AFTER) { update_archived_status }
   • def update_archived_status
   • batch.update_all(archived: project.archived)

4. ee/app/workers/sbom/sync_archived_status_worker.rb

   • class SyncArchivedStatusWorker
   • ::Sbom::SyncArchivedStatusService.new(event.data['project_id']).execute

5. ee/app/models/sbom/occurrence.rb

   • scope :unarchived, -> { where(archived: false) }
   • archived: project.archived,

6. ee/app/finders/sbom/aggregations_finder.rb

   • Sbom::Occurrence.for_namespace_and_descendants(namespace).unarchived.loose_index_scan(column: :traversal_ids)
   • .unarchived
   • .unarchived