## Overview This epic addresses the implementation of a VulnerabilitiesOverTime chart that shows **active vulnerabilities over time** (vulnerabilities in `detected` or `confirmed` states) using historical state data, rather than just vulnerabilities created per day. ## Problem Statement Currently, the ES-based vulnerability charts can only display vulnerabilities created each day, not those that remain active on a given date. The core issue is that we do not store state changes over time, which prevents us from determining the historical state of vulnerabilities on any given date. Specifically, we lack historical state transition data to reconstruct what vulnerabilities were "active" on past dates Note: The existing PG-based vulnerability statistics only count vulnerabilities in active states (`detected`, `confirmed`), and we need to maintain this behavior for consistency. ## Requirements ### Chart Specifications - **Level**: Project or Group - **Time parameters**: - Period: 30, 60, or 90 days - Interval: 1 day - **Filtering**: - Severity - Report Type - Project (only for Group level) - **Grouping**: - Severity - Report type ## Technical Challenges ### Current Limitations 1. No State Change Tracking: The primary blocker is the absence of historical state data in ES 2. State History Reconstruction: Need to track vulnerability state changes over time to determine what was "active" on any given historical date 3. Active State Filtering: Must filter for only `detected` and `confirmed` states, matching existing PG behavior 4. Performance: ES queries for historical state reconstruction need to be optimized 5. Storage: Storing historical state data efficiently without excessive document size growth ### Data Inconsistency Issue The backend API doesn't return severities with zero counts. For example, a day might only return `UNKNOWN` and `LOW` severities, requiring frontend handling of missing severity data points. ## Proposed Solutions ### Option 1: Daily State Field - Add `state_at_dates` field to ES documents storing daily state snapshots - Run daily cron job to update vulnerability states - Direct field lookup for historical queries - Filter for active states (`detected`, `confirmed`) on each date - **Pros**: Fast queries, direct ES aggregation - **Cons**: Storage overhead (potentially unlimited due to retention policy resets), daily maintenance job ### Option 2: State History with Application Processing (Recommended) - Store `state_history` array with state transitions and timestamps - Query ES for state history, process in application to determine state on target dates - Filter for active states in application logic - **Pros**: Storage efficient, only stores changes, no daily maintenance job - **Cons**: Complex queries, application processing overhead, more bug-prone ### Considerations for Both Options - **Primary vs Secondary Data Store**: ES is secondary to PG, so disaster recovery implications must be considered - **Audit Table Integration**: Leverage existing `vulnerability_state_transitions` PG table for state changes - **Severity Transitions**: May need to track severity changes over time for filtering - **Document Size Impact**: Coordinate with Global Search team on ES index optimization - **Active State Definition**: Only count vulnerabilities in `detected` or `confirmed` states