Namespace level statuses for DB based settings

The SD and CS have project-level database persistent security settings stored under ProjectSecuritySetting, namely Secret Push Protection (SPP) and Container Scanning for Registry (CSfR). For project-level analyzer coverage, we currently use these fields as-is (see issues 1, 2, 3).

However, for Group-level analyzer statuses, we currently don't have this data available. What we need is a counter representing the number of projects within a namespace where SD or CS is enabled. Specifically, SD is considered enabled when SPP is on or when a pipeline has a secret detection job. Similarly, CS is enabled when CSfR is on or when a pipeline includes a container scanning job.

Suggested Solution:

  1. Add new analyzer_types and separate SPP and CSfR into distinct pills in the security inventory UI:
    This would make the data we already have more accurate and allow for a clear separation between pipeline-based data and DB setting-based counters. We would need to introduce types for SPP and CSfR and backfill the data based on the existing security settings. Additionally, we should ensure that this data is maintained and its changes are propagated up to project ancestors on every setting update.

  2. Add update logic to determine the actual counter:
    Since we only need a counter, we can implement logic to check whether a counter update is necessary for a namespace based project settings and pipeline jobs. This logic should be triggered after a pipeline execution and when a setting was changed in the database for SPP and CSfR. For example, the number of SD-enabled projects for a given project ancestor should not decrease as long as SPP is enabled, even if the latest pipeline didn’t contain any SD jobs. This approach will maintain the current types, but the SD and CS types will reflect an aggregated count based on calculations and logic that take into account the DB settings.

    Key changes:

    1. Adjustment service: The AnalyzerNamespaceStatuses::AdjustmentService should take project_setting into account when calculating SD and CS statuses.
    2. Post pipeline update service: The AnalyzersStatus::UpdateService logic should consider the existing project_setting values before propagating diffs to project's ancestors.
    3. Setting SPP and CSfR values:: All the existing places that set these values should trigger AnalyzerNamespaceStatus records creation. This potentially includes creating a new project.
    4. Project and group events handlers: Existing project and group events handlers should consider project setting when updating and adjusting counters. This might already be taken care of by changing the AdjustmentService.

  3. Add new aggregated types (based on solution 2, but with new types):
    Instead of using the existing secret_detection and container_scanning types to hold the aggregated counters for groups (like in solution 2), we can introduce new types for these aggregated counts. This way we can keep a correlation between the project-level counters and the group-level counters of the same existing types, while having to implement the same logic parts suggested in solution 2. This can also potentially enhance the granularity level of our data, allowing for easier separation of setting-based and pipeline-based counters in the future.

    Key changes:

    1. Add new analyzer types: Add new analyzer_types: secret_push_protection, container_scanning_for_registry, aggregated_secret_detection and aggregated_container_scanning. Those types should be used for AnalyzerNamespaceStatuses only, while the two latter will hold the number of projects covered with either pipeline or setting-based security tools.
    2. Adjustment service: The AnalyzerNamespaceStatuses::AdjustmentService should maintain those new types while taking project_setting into account when calculating.
    3. Post pipeline update service: The AnalyzersStatus::UpdateService logic should consider the existing project_setting values before propagating aggregated_container_scanning or aggregated_secret_detection to project's ancestors.
    4. Setting SPP and CSfR values: All the existing places that set these values should trigger calculation of AnalyzerNamespaceStatus with the new relevant type. The aggregated types might need to be updated based on the existing project-level statuses. This potentially includes having records created when creating a new project.
    5. Project and group events handlers: Existing project and group events handlers should consider project setting when updating and adjusting counters. This might already be taken care of by changing the AdjustmentService.

  4. Add new analyzer types to project only (based on solution 2 and 3)
    This approach builds on suggestion 3. With this solution, we will be storing the new analyzer type records at the project level, in addition to the existing analyzer types (for example, a project will have 3 analyzer status records related to secret detection: general secret detection, secret push protection, pipeline based secret detection). In this example, the general secret detection status will feed into the group level status for secret detection. Storing the setting based records in the analyzer status table means we will be able to avoid a number of calls to the project_security_settings table when we need to determine the status for general secret detection for a project and it's group(s).

    Key changes:

    1. Add new analyzer types: Add new analyzer_types: secret_detection_secret_push_protection, secret_detection_pipeline_based, container_scanning_for_registry, and container_scanning_pipeline_based. Those types should be used for AnalyzerStatuses only, while the existing secret_detection and container_scanning will hold the number of projects covered with either pipeline or setting-based security tools.
    2. Adjustment service: No changes to the AnalyzerNamespaceStatuses::AdjustmentService will be required, accept possibly to exclude calculating the new analyzer types.
    3. Post pipeline update service: The AnalyzersStatus::UpdateService logic should consider the existing project_setting values before propagating container_scanning or secret_detection to project's ancestors.
    4. Setting SPP and CSfR values: All the existing places that set these values should trigger calculation of AnalyzerStatus with the new relevant type. The aggregated types might need to be updated based on the existing project-level statuses. This potentially includes having records created when creating a new project.
    5. Project and group events handlers: Existing project and group events handlers should consider project setting when updating and adjusting counters.
    6. Backfill migration Currently, our records for secret_detection and container_scanning only represent the status of pipeline based scans. We will need to backfill records so that AnalyzerProjectStatus contains 3 records, as well as re-calculate AnalyzerNamespaceStatus to properly reflect the updated count based on pipeline-based data and DB setting-based counters.
Edited by rossfuhrman