Agree on the success metrics for the v3 package metadata syncs

Follow-up from the review of !252822 (merged), where @nilieskou pointed out that the v3 sync instrumentation was built without first agreeing on what we want to track and how we measure success.

This issue collects what already ships, what was asked for and is still open, and is the place to have that discussion before more events are added.

What ships in !252822 (merged)

One internal event per (data type, outcome) pair, for both v3 datasets (licenses, malware_advisories). Event names are sync_pmdb_v3_<data_type>_<outcome>, label carries the purl_type, and the allowlist lives in V3SyncService::SYNC_OUTCOMES.

Run outcomes:

  • full_completed / delta_completed - a run read every file and persisted every slice
  • full_interrupted / delta_interrupted - a run stopped on the stop signal, checkpoint left at the last complete unit
  • full_not_persisted / delta_not_persisted - every file read but a slice failed to persist, so the checkpoint did not advance
  • full_failed / delta_failed - the run raised and aborted before reaching an outcome
  • delta_up_to_date - a delta run found no new archives, so the registry is already current

Registry and run-level outcomes:

  • aborted_no_token - the run stopped before any registry because the instance token (IJWT) was unavailable
  • skipped_stop_signal - a registry skipped before its own sync started, budget already spent
  • bulk_delta_skipped_stop_signal - a registry queued for the shared bulk /delta skipped before the fetch
  • bulk_delta_failed - the shared bulk /delta fetch raised, dropping every queued registry
  • skipped_unsupported - a registry dropped because PDS /supported does not serve it

Measurements:

  • delta_backlog - how many delta archives PDS offered for a registry, carried in value. Zero means fully caught up; PDS caps a response at 20, so a registry at the cap is behind by at least that much.

Every event carries the GitLab Standard Context, which includes instance_id, unique_instance_id, host_name, instance_version and deployment_type, so a single instance can be isolated without an extra field.

Derivable from the above

  • How often an instance interrupts a full upsert and resumes - count full_interrupted per instance and registry
  • How many runs it takes to finish an upsert - full_interrupted events between two full_completed for the same instance and registry
  • The same for delta - delta_interrupted between two delta_completed

Still open

  • Agree on which of these are the success metrics for the v3 syncs, and what the target values are
  • Decide whether dashboards or alerts are built on these events, and who owns them
  • Confirm the malware advisory outcomes are complete (raised by @nilieskou for @bala.kumar on the MR)
  • Decide whether the Prometheus and Service Ping work described in #602417 is still wanted alongside these events, given internal events cannot alert on a sync that stopped firing

/cc @onaaman @bala.kumar

Edited by Ahmad Zaydan