Create a malware exporter

Proposal

The refinement of this issue depends on Spike: PackageMetadata sync data format v3 (#584273 - closed)

The main idea is that the exporter will have an execution mode for malware advisories for which it will read data from the malware_db_advisory table and export them in a v3 format in a private bucket.

The format of the data needs to be agreed. Also due to the v3 format the exporter will most probably be required to take additional steps including compression.

Requirements

• When updating manifest.json file in v3 in the bucket we need to make sure that the file is not being downloaded by instances.
• We need to keep a schema and version it with SchemaVer. See #587162 (comment 3064506944). Schema should be stored in https://gitlab.com/gitlab-org/security-products/security-report-schemas
• We need a module that translates OSV versions into NPM and other PURL type expressions. We can use this algorithm. See #587162 (comment 3115896155). Read OSV spec docs.
• Before exporting a JSON object make sure it is validated against the schema

Suggested Schema

Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "GLAM Malware Advisory",
  "type": "object",
  "required": ["malware_advisory", "packages"],
  "properties": {
    "malware_advisory": {
      "type": "object",
      "required": ["id", "source", "title", "description", "published_date", "identifiers"],
      "properties": {
        "id": {
          "type": "string",
          "description": "Unique identifier for the malware advisory",
          "pattern": "^GLAM-[0-9]{4}-[0-9]{2}-[0-9]{5,}$"
        },
        "source": {
          "type": "string",
          "description": "Source of the advisory",
          "const": "glam"
        },
        "title": {
          "type": "string",
          "description": "Title of the malware advisory"
        },
        "description": {
          "type": "string",
          "description": "Detailed description of the malware"
        },
        "published_date": {
          "type": "string",
          "description": "Date when the advisory was published",
          "format": "date"
        },
        "withdrawn": {
          "type": "string",
          "description": "Optional date when the advisory was withdrawn",
          "format": "date"
        },
        "identifiers": {
          "type": "array",
          "description": "List of identifiers for this malware advisory",
          "minItems": 1,
          "items": {
            "type": "object",
            "required": ["type", "name", "value", "url"],
            "properties": {
              "type": {
                "type": "string",
                "description": "Type of identifier (e.g., GLAM, MAL)"
              },
              "name": {
                "type": "string",
                "description": "Name/display value of the identifier"
              },
              "value": {
                "type": "string",
                "description": "Actual identifier value"
              },
              "url": {
                "type": "string",
                "description": "URL reference for the identifier (may be empty)"
              }
            }
          }
        }
      }
    },
    "packages": {
      "type": "array",
      "description": "List of affected packages",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["name", "purl_type", "affected_range"],
        "properties": {
          "name": {
            "type": "string",
            "description": "Name of the affected package"
          },
          "purl_type": {
            "type": "string",
            "description": "Package URL type (e.g., npm, pypi, maven)"
          },
          "affected_range": {
            "type": "string",
            "description": "Version range affected by the malware"
          }
        }
      }
    }
  }
}

Proposal for breaking down work

• We need a module that parses the OSV advisory affected versions and gives a PURL specific type range (semver_dialect).
• We need to be able to always extract a full_data.zst file. Extend the glam-exporter with this capability. This needs to be a module that can be reused in the future by other exporter types.
• We need to add all the functionality that we have for public advisories but now also for malwares. Ignoring ful_data.zst.
• We need to write in the bucket the manifest.json file. We can start doing that for all data. Not only for malware.
• Create a scheduled job for exporter
• Update Deployment documentation, scripts and test E2E