# Problem ## Description When scans change state from `enabled` to `not enabled` in a project, the Security Inventory hides the data and related links that help users navigate to understand vulnerabilities in associated projects. In spite of the fact that security scanners had run previously, and that vulnerabilities were populated for the project, from the inventory, historical scan data becomes invisible in security dashboards (data still exists, just not visible). This impacts the ability to track security scan configuration requirements. ## What data is hidden? When a previously configured analyzer was not run on the main branch during last pipeline then it is shown as "Not enabled". This hides some UI elements: * Pipeline link in the Popup over the Project Analyzers Statuses * Information about failed / succeeded scanners when they were not run in the last pipeline because they turn to Not configured ## Reproduction steps Given a project with Security Inventory showing enabled statuses. I used a simple Dockerized rack app that shows a hello world HTML page with auto dev ops enabled. Push a change, wait for it to complete the scans. Observe the scan statuses report actual Analyzer statuses ( success or failures ) ### Via skipping a scan on custom CI run 1. Go to Build → Pipelines → Run pipeline. 2. Under Variables, add the variables ( see below ). 3. Click Run pipeline. 4. Once finished, observe Sec Inventory and Sec testing show "Not enabled" for the scans. ``` SAST_DISABLED=true SECRET_DETECTION_DISABLED=true DEPENDENCY_SCANNING_DISABLED=true CONTAINER_SCANNING_DISABLED=true DAST_DISABLED=true ``` ### Via DAST on demand scan 1. Create On demand dast scan and run it. 2. After DAST completes - observe all the statuses apart from DAST are "Not enabled". 3. Trigger Pipeline ( Go to Pipelines \> New Pipeline ), wait until they complete 4. Observe the Inventory statuses are shown again # Solution When an analyzer was not run during on main branch pipeline * The inventory should preserve the last known scanner state \[https://gitlab.com/gitlab-org/gitlab/-/work_items/578392#note_3093691957\\\\\\\] * The analyzer status transitions to Stale after 3 consecutive times when it was not run \[https://gitlab.com/gitlab-org/gitlab/-/work_items/578392#note_3210632331, https://gitlab.com/gitlab-org/gitlab/-/work_items/578392#note_3125391114\\\\\\\] # Tech approach We will introduce the Stale status to the `AnalyzerProjectStatus` which will be set by the `UpdateService` in line with existing implementation. To track consecutive runs without an analyzer we will introduce `consecutive_absence_count` column. And `stale` column on the `analyzer_namespace_statuses` . Track consecutive bypasses via a counter on `analyzer_project_statuses`. On each default-branch pipeline completion: - Analyzer **ran** - upsert new status, reset `consecutive_absence_count = 0` - Analyzer **absent** (pipeline-tracked types only) - `consecutive_absence_count += 1`; if count reaches threshold (3) → transition to `:stale` The proof of concept is here: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/230019/diffs#diff-content-c02999e3d025ef7dc7b0ff7fc6ffc3bf24af7b70 # Delivery plan You can see the full detailed overview in this comment: https://gitlab.com/gitlab-org/gitlab/-/work_items/578392#note_3228613587 ## Story 1 - Analyzer status becomes "Stale" after it was not run 3 consecutive times ( this issue ) Tasks: ### 1. Introduce Backend fields https://gitlab.com/gitlab-org/gitlab/-/work_items/596019 * [x] Migrations * [x] Add `consecutive_absence_count integer NOT NULL DEFAULT 0` to `analyzer_project_statuses` * [x] Add `stale bigint NOT NULL DEFAULT 0` to `analyzer_namespace_statuses` * [x] Add `:stale` to `Enums::Security.analyzer_statuses` * [x] Add validation for greater than 0 for `:stale` in `AnalyzerNamespaceStatus` model * [x] Update `ee/spec/factories/analyzer_namespace_statuses.rb` and failing specs * [x] Expose stale status on `AnalyzerProjectStatusType` ### 2. (Backend) Update stale status related fields https://gitlab.com/gitlab-org/gitlab/-/work_items/596020 * [ ] Update the Services * [ ] `DiffService` returns `stale` count * [ ] `AncestorsUpdateService` updates `stale` count * [ ] `UpdateService` upserts the `stale` count to projects and passes the stale count `AncestorsUpdateService` * [ ] `AdjustmentService` accounts for the `stale` count * [ ] When `AnalyzerProjectStatus` is stale it still has to keep the reference to the last job it run, and date when it run, so that AppSec team could investigate it. * [ ] Introduce a feature flag `security_inventory_stale_analyzer_status` ( type: `wip` ) * [ ] If the `security_inventory_stale_analyzer_status` is enabled then the `AnalyzerNamespaceStatus#not_configured` accounts for `stale` in the calculation. * [ ] Update GraphQL types * [ ] Expose stale count on `AnalyzerNamespaceStatus` ### 3. (Frontend) show `stale` status on Security Inventory https://gitlab.com/gitlab-org/gitlab/-/work_items/596022 See mockups from the comment above: https://gitlab.com/gitlab-org/gitlab/-/work_items/578392#note_3197865749 * [ ] Project level: when project status is `STALE` ( read from `projects.analyzerStatuses.status`) * [ ] Show "Stale" status * [ ] Show icon * [ ] When the status is stale UI still has to display a link to the last job run, and date when it run, so that AppSec team could investigate it. * [ ] Group level: when the `security_inventory_stale_analyzer_status` is enabled * [ ] Query `groups.analyzerStatuses.stale` field in the `SubgroupsAndProjects` query * [ ] Add the coloured segment representing Stale count. The colour should match the Stale status icon * [ ] Add Stale line with the number to the tooltip ## Story 2 - Filtering by stale status As an infosec engineer, I want to filter the Security Inventory by Stale status so that I could further investigate and fix any potential configuration issues https://gitlab.com/gitlab-org/gitlab/-/work_items/596024 ## Remove the feature flag After the `security_inventory_stale_analyzer_status` was released to all customers we need to clean up the conditional logic related to the feature flag. https://gitlab.com/gitlab-org/gitlab/-/work_items/596026