Add metrics for the v3 syncs (malware advisories and licenses)

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Scope update (metrics only). Lifecycle and ingestion logs already exist, and !251813 (merged) generalizes them so licenses get the same vocabulary. This issue is metrics only, for both v3 datasets: malware advisories and licenses. Validation-skip counts: #607505. Wave 4 of epic &20876.

Why

No metrics exist anywhere in the sync path. Sidekiq worker metrics count a skipped run (lease taken, flag off, jitter re-enqueue) as success, so a registry silently falling behind is invisible. Licenses and malware advisories share the v3 service and connectors after #622511 (closed), so one instrumentation pass covers both.

What

In priority order:

  1. Sync lag per registry, via a cron gauge exporter reading pm_checkpoints:
    • pm_sync_lag_seconds{data_type, purl_type, version_format}: now - to_timestamp(sequence)
    • pm_checkpoint_stale_seconds{...}: now - updated_at
    • Exporter, not in-worker: lag must keep reporting when the sync worker is wedged.
  2. Outcome counters in V3SyncService, from values the logs already compute:
    • pm_sync_runs_total{data_type, sync_mode, phase}
    • pm_sync_records_ingested_total{data_type, purl_type}
  3. PDS request outcomes in the Pds base, inherited by both subclasses:
    • pm_pds_requests_total{endpoint, status}
    • Archive download / decompression failure counters
  4. Storage-mode Service Ping: storage_type per data_type (pds / offline / gcp). Prometheus never leaves an air-gapped network, so the offline adoption split (#616870 (closed), #594758 (closed)) needs Service Ping.

Internal events already cover which code path a sync took (sync_pmdb_v2_*, !237975 (merged)). They fire per call and land in Snowflake, so they cannot alert on a sync that stopped firing. Prometheus covers health, lag and volume; Service Ping covers the storage-mode split.

Non-goals: per-shard metrics (cardinality), dashboards/alerts (follow-up), IngestedMalwareAdvisoryEvent counts (unless CVS asks), duplicating the existing path-selection events.

How

  • Instrument V3SyncService and Connector::Pds once, after #622511 (closed) merges. Labels: data_type, purl_type, version_format only.
  • Lag exporter: cron worker over pm_checkpoints (sequence, updated_at, full_sync_target_sequence for bootstrap-in-progress).
  • Service Ping metric next to the existing package_metadata_sync_activation_by_type.

Prior art

Signal Precedent
Lag exporter Geo::MetricsUpdateWorker / Geo::MetricsUpdateService (!3187 (merged)), named the reference pattern in prometheus_metrics.md; also gitlab_ghost_user_migration_lag_seconds
Outcome counters job_trace_archive_failed_total, loose_foreign_key_processed_deleted_records, sbom_schema_report_export_validation_failures_total
Request counters gitlab_openbao_requests_total{operation, method, outcome} + duration histogram
Mode Service Ping settings.package_metadata_sync_activation_by_type_metric (!146176 (merged)), database.mode (!115900 (merged))
Label scheme !237975 (merged) (#600562 (closed)) instrumented the v2 Gcp connector with one internal event per data_type, purl_type as the label, path key as the property, and no label where purl_type is nil. Same dimensions, same nil carve-out

Register new metrics in gitlab_metrics.md; Service Ping key paths per metrics_dictionary.md.

Related

Edited by Ahmad Zaydan