Continous updates for change data on Vulnerabilities Index
What does this MR do and why?
Continuous updates sync for change data on Vulnerabilities Index.
In this MR:
- The change data during
VulnerabilityandVulnerabilities::Readmodel during an create, update and delete operation are synced to ES. - Vulnerability creation or updation from the bulk ingestion process during pipeline scans or CVS is also synced to ES.
This is the primary change data capture and there are other low frequent cases (#534622 (closed), #536534 (closed), #535117 (closed)) which are tracked in the other issues and will be addressed respectively.
This change is behind a FF, till the other sync cases are addressed it can be selectively enabled on the test projects/groups on staging and production.
References
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
Validate bulk ingestion ES sync
-
Ensure that ES is setup in local and the earlier vulnerabilities Index is created.
-
Ensure that local setup is running SASS mode steps here, Enable the FF globally in rails console
Feature.enable(:vulnerability_es_ingestion) -
Import the project from here into local using the import by url option.
-
(Optional) Truncate the index:
curl -XPOST "http://localhost:9200/gitlab-development-vulnerabilities/_delete_by_query" -H "Content-Type: application/json" -d'
{
"query": {
"match_all": {}
}
}'
-
In the imported project, run pipeline on the master branch and allow the pipeline to complete. This will seed the vulnerabilities data and also populate ref queue in Sidekiq for the ES bookkeeping. The bookkeeping Sidekiq cron runs every 1 minute.
-
After a minute (or run bookkeeping manually in rails console
Elastic::ProcessBookkeepingService.new.execute) query the index to view the vulnerabilities data being populated. Curl command to query is available below:
curl -XGET "http://localhost:9200/gitlab-development-vulnerabilities/_search" -H "Content-Type: application/json" -d'
{
"query": {
"match_all": {}
},
"size": 10000
}'
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #515565 (closed)