Security dashboard - Exclude No Longer Detected by adding undetected_since field to vulnerability PG and ES

What does this MR do and why?

This MR:

• Updates the application logic to create a new detection transition record
• Add ES migration to add undetected_since field
• Update ee/lib/search/elastic/references/vulnerability.rb with new schema version and field setup
• Add the field in ee/lib/search/elastic/types/vulnerability.rb
• ES sync is triggered when undetected_since is updated in which is already done as part of the Vulnerabilities::DetectionTransitions::InsertService class

Previous MRs:

• Add detection_transitions table to track no lon... (!214443 - merged) • Subashis Chakraborty • 18.7
• Add service to insert detection transitions (!214890 - merged) • Subashis Chakraborty • 18.7

Related to #578567 #578566 #578568

References

• Closes #578567
• Closes #578568
• Blocks #582985
• Related to #578566

Validation steps

Step 1: set up

1. Make sure you have Elasticsearch running locally. You can follow this guide: https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/elasticsearch.md#setup

2. Start Rails console by running rails c

Step 2: Create test data

1. Enable the feature flag by running

Feature.enable(:new_security_dashboard_exclude_no_longer_detected)

4. Run:

project = Project.find xx  # Use the ID of a project on your local dev
finding = project.vulnerability_findings.first
vulnerability = finding.vulnerability

Step 3: Verify DetectionTransition in PG

1. Create a detection transition with detected: false:

transition = Vulnerabilities::DetectionTransition.create!(
  vulnerability_occurrence_id: finding.id,
  project_id: finding.project_id,
  detected: false,
  created_at: 2.days.ago
)

finding.reload!
finding.latest_detection_transition.detected  # Should be false

Step 4: Verify undetected_since is indexed in Elasticsearch

1. Check that the ES migration adds the undetected_since field mapping by running the following command in the terminal:

curl -s "http://localhost:9200/gitlab-development-vulnerabilities/_mapping?pretty" | grep -A 5 "undetected_since"

Expected output:

   "undetected_since" : {
          "type" : "date"
        },
        "updated_at" : {
          "type" : "date"
        },

2. In the rails console run:

Vulnerabilities::EsHelper.sync_elasticsearch([vulnerability.id])

3. Check if undetected_since is populated by running this in your terminal:

curl -s "http://localhost:9200/gitlab-development-vulnerabilities/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "term": {
        "vulnerability_id": {
          "value": xxx # your vuln id
        }
      }
    },
    "_source": ["vulnerability_id", "undetected_since"]
  }'

Expected output: undetected_since should be populated.

Step 5: Verify undetected_since is not populated for detected vulnerabilities

1. In your rails console, run:

finding3 = project.vulnerability_findings.third
Vulnerabilities::DetectionTransition.create!(
  vulnerability_occurrence_id: finding3.id,
  project_id: finding3.project_id,
  detected: true,
  created_at: 1.day.ago
)

Vulnerabilities::EsHelper.sync_elasticsearch([finding3.vulnerability.id])

finding3.vulnerability.id # use this ID for the next step

2. In your terminal run:

curl -s "http://localhost:9200/gitlab-development-vulnerabilities/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "term": {
        "vulnerability_id": {
          "value": xxx # the ID you got in the previous step.
        }
      }
    },
    "_source": ["vulnerability_id", "undetected_since"]
  }'

Expected: undetected_since should be null

Step 6: Check with disabled feature flag:

Feature.disable(:new_security_dashboard_exclude_no_longer_detected)

finding2 = project.vulnerability_findings.second
Vulnerabilities::DetectionTransitions::InsertService.new([finding2], detected: false).execute

Vulnerabilities::DetectionTransition.where(vulnerability_occurrence_id: finding2.id).count # Should be 0

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.