SPIKE: Investigate how to track and get notified about errors in Package Metadata DB (License DB)

Topic to Evaluate

• Link to other Issue

Tasks prior to evaluation

• Understand the infrastructure we have for collecting errors, metrics and alerts. Relates to #427445 (closed).
  • Errors
    • Sentry
    • Integrated error tracking
  • Metrics
    • Export License-DB GCP metrics to license-db Pro... (#427445 - closed) • Nick Ilieskou • 16.6 • On track
  • Alerts
    • Slack - alerts have been configured for #g_secure-composition-analysis-alerts.

Outcome

• Identify how to track errors that occur in License DB (AKA Package Metadata DB) services. Perhaps custom metrics?
  • Integrated error tracking
• Identify how to notify Secure Composition Analysis about these errors.
  • Slack - alerts have been configured for #g_secure-composition-analysis-alerts.
• Create implementation issues.
  • See #416158 (comment 2142075359)

Risks and Implementation Considerations

We should update the runbooks and reaction rotation documentation so that we check these proactively in addition to the alerts we receive.