## TL;DR Exclude no longer detected vulnerabilities from the `VulnerabilitiesOverTime` chart ## Scope ### In Scope What is included in this deliverable. - [x] Add `detection_transitions` table to track `undetected_since` (PostgreSQL + ES) - [x] Sync the `undetected_since` to ES when vulnerabilities are marked as resolved - [x] Modify the `VulnerabilitiesOverTime` query to filter out vulnerabilities after their resolution date - [x] Backfill undetected_since with timestamp during backfill ### Out of Scope - [ ] Backfilling historical `undetected_since` data (not possible to determine past resolution dates) - [ ] Modifying any other new Security Dashboard chart other than `VulnerabilitiesOverTime` ## Problem Statement The Vulnerabilities Over Time chart currently includes "No longer detected" vulnerabilities in its counts indefinitely, which inflates the numbers and prevents accurate security trend tracking. Users cannot see the true state of their security posture over time because resolved vulnerabilities continue to be counted after they're no longer present. ## Current Timeline / Action Items 1. [x] Globally enable `new_security_dashboard_exclude_no_longer_detected` (by Friday 30 Jan - @lorenzvanherwaarden) 2. [x] 2 migrations as 2 MRs: 1. [x] Truncating the transitions table (merged by Friday 30 Jan - @bala.kumar) 2. [x] Reintroduce the earlier backfill to populate transitions table. Merging this MR is blocked on the first MR and its timestamp should be after the first MR. (merged by Tuesday 3 Feb - @charlieeekroon) 3. [x] Teleport Rails console verification [**verify-historical-no-longer-detected-backfill**](https://gitlab.com/gitlab-org/govern/threat-insights-demos/verification-projects/exclude-no-longer-detected-verification/verify-historical-no-longer-detected-backfill) (by Friday 6 Feb - @bala.kumar) 4. [x] ES backfill migration for `undetected_since` field using [MigrationDatabaseBackfillHelper](https://docs.gitlab.com/development/search/advanced_search_migration_styleguide/#searchelasticmigrationdatabasebackfillhelper) (by Friday 6 Feb - @charlieeekroon) 5. [x] After the ES backfill, enable `new_security_dashboard_over_time_chart_undetected_since` to verify historical vulnerabilities on an older and large project like **gitlab.com**. ES backfill is dependent on the BBM completion; for code reference we can refer to the risk score ES migration. Max duration of 2 days run time. (by Monday 9 Feb – @charlieeekroon) 6. [x] `new_security_dashboard_over_time_chart_undetected_since` full rollout and default both feature flags (by Wednesday 11 Feb) Project: [**verify-historical-no-longer-detected-backfill**](https://gitlab.com/gitlab-org/govern/threat-insights-demos/verification-projects/exclude-no-longer-detected-verification/verify-historical-no-longer-detected-backfill) – verification project (before migrations have run without ffs enabled) with multiple no longer detected vulnerabilities that we don't touch until after migrations and backfill * feature flag for creating DetectionTransition records: `new_security_dashboard_exclude_no_longer_detected` * feature for excluding no longer detected in chart: `new_security_dashboard_over_time_chart_undetected_since` (This comment is copy-pasted from https://gitlab.com/groups/gitlab-org/-/epics/19780#note_3047878014) ## Feature Flag - [x] Required\ **Flag name:** `new_security_dashboard_exclude_no_longer_detected`\ **Default state:** Disabled\ **Rollout plan:** Optional gradual rollout details. ## Dependencies See _Tasks_ section for detailed dependency chain. ## Implementation Notes * No backfill needed as we cannot determine historical resolution dates * ES sync happens immediately when field is set (no manual reindexing required) ## Resources - Documentation: \[link\] - Related issue: #471269