Fix Secrets Manager observability issues

Jayakrishnan Mallisseryrequested to merge
jay-sm-observability-rails-fixes into master

Why this MR ?

  • Part of #611251
  • Six provisioning and deprovisioning workers declare urgency :high but make 14+ sequential calls to OpenBao, so the 10s apdex target was never realistic — (the Sidekiq SLO alert would go permanently red once enrollment picks up. FYI, urgency is a Sidekiq worker attribute declared with the urgency class macro app/workers/concerns/worker_attributes.rb:81-85, valid values :high, :low, :throttled). IMPORTANT: It is not a scheduling priority in the queue sense — it is a declared performance contract, and the numbers behind that contract live in lib/gitlab/metrics/sidekiq_slis.rb:6-19
  • The audit-log worker and the two maintenance cron workers logged nothing identifying the customer, so no Secrets Manager incident could be narrowed to one project or group in Kibana
  • The OpenBao request metrics recorded the wrong outcome in both directions, and the duration histogram had no outcome label at all, so a failing OpenBao actually looked fast
  • This MR fixes all three 🙈

What does this MR do ?

  • Set urgency :low and add worker_has_external_dependencies! on the six workers
  • Push project and namespace context onto the audit-log worker and the two maintenance cron workers, drop two Scalability/CronWorkerContext disables
  • Move the conflict handling and response validation inside the instrumented block so the recorded outcome matches reality
  • Add outcome to the duration histogram, plus a new gitlab_openbao_request_errors_total counter keyed by fault type
  • Fix the troubleshooting doc, which named worker classes that are never enqueued, and document the new and changed metrics

Notes for review

  • urgency :low brings these workers under Sidekiq/EnforceDatabaseHealthSignalDeferral, which urgency :high was exempt from
  • Only real behaviour change here: dropping to urgency :low cuts the concurrency limit for these workers from 0.35 to 0.25 of shard capacity, and on GitLab.com it moves them to a different Sidekiq fleet. Net effect is slower provisioning under shard pressure.
  • urgency :low and worker_has_external_dependencies! have to land together — every_sidekiq_worker_spec.rb fails a worker that is both high urgency and external, so splitting them into two commits would break the build in between.

References

Edited by Jayakrishnan Mallissery

Merge request reports