Extend advisory-feeder to ingest latest changes from trivy-db (using a cursor)

Goal

The goal is to extend the advisory-feeder so that it checks if there are any new changes in the trivy-db. This is done by using a cursor. A cursor is a file in the internal gcp bucket where we store the last processed trivy-db image. The advisory-feeder will download the latest db file and compare the differences with the last processed db file. Only these changes will be sent to the advisory-processor.

Useful Links

• https://gitlab.com/adamcohen/trivy-db-tools/-/blob/main/main.go?ref_type=heads
• #422869 (comment 1522612583)
• Extend advisory-feeder to ingest all trivy-db d... (#423239 - closed)
• Trivy-db advisory structure fields

Requirements

• For this issue we need --internal-bucket that specifies the bucket that contains the cursor.
  • The path of the cursor looks like this: dev-internal-bucket-001ade7d553dbeb1/advisory_feeder/trivy-db/cursor.txt
  • The content of the cursor: YYYYMMDDHH which represents the tag of the latest trivy-db. Example of db to pull: registry.gitlab.com/gitlab-org/security-products/dependencies/trivy-db-glad:2023082703
• Trivy-db files are download using trivy-db-glad.

Implementation Plan

Advisory feeder project

• In the Read the cursor. If the cursor doesn't exist we do a full ingestion which is already implemented by Extend advisory-feeder to ingest all trivy-db d... (#423239 - closed). If the cursor is present we download the last processed trivy-db.
• Using the Gitlab Registry API find the latest image tag and download it using oras.
• Use the bbolt library to read the trivy.db files.
• Compare the two files and make a list of all the differences. The latest db file always win. We ignore deletion. We only care about adding or modifying os packages and vulnerabilities. Application packages are also ignored.
• Send the new changes using pubsub
• Add a context timeout to ensure that pubsub can timeout. Relates to https://gitlab.com/gitlab-org/security-products/license-db/license-feeder/-/merge_requests/52#note_1552801872
• Add unit tests
• Add a section in the readme file explaining the basics around advisory ingestion.
• Create a release

Deployment project

• Update advisory-ingestion.drawio.png by adding trivy-db as a source.
• Update security.md on a need basis.
• Update Feeder testing instructions on a need basis.
• Update Creating scheduled pipelines for license-feeder with the new flags that have been added.
• Deploy on dev and test.
• Deploy on prod
• Create scheduled jobs for dev for advisory-feeder using trivy-db as a source.
• Create scheduled jobs for prod for advisory-feeder using trivy-db as a source.
• Update existed scheduled jobs with the new flags