Draft: Add Sidekiq observability for organization maintenance mode skip/cancel/filter events

What does this MR do and why?

Implements structured logging and Prometheus metrics for every Sidekiq job that is skipped, cancelled, or filtered because its organization is in maintenance mode. This is part of #602826 (closed) (Org Read-Only Mode: observability for skip/cancel/filter events), which is part of the epic gitlab-org#20404 (Implement Organization Maintenance Mode).

New components

Gitlab::SidekiqLogging::OrganizationMaintenanceModeLogger (singleton, follows PauseControlLogger/DeduplicationLogger patterns):

  • Emits a structured log entry to the Sidekiq logger for each intercepted job
  • Payload fields: organization_maintenance_mode.action (skipped|cancelled|filtered), organization_maintenance_mode.organization_id, class, jid, job_status, pid

Gitlab::SidekiqMiddleware::OrganizationMaintenanceMode::Server (server-side middleware):

  • Placed after CurrentOrganization::Server so Current.organization is already set
  • Cron workers (queue_namespace == 'cronjob') in maintenance -> skip (raises Sidekiq::JobRetry::Skip, not re-enqueued)
  • Org-scoped workers with maintenance_enforced? -> cancel (dropped without retry)
  • Org-scoped workers in maintenance but without enforcement -> filter (logged, job proceeds)
  • Each action increments sidekiq_organization_maintenance_mode_jobs_total Prometheus counter (labels: worker, action)

Design rationale

The single log/metric signal is shared by:

  • Steady-state observability dashboards and alerts
  • The cutover-readiness endpoint (drain completion check)

This means operators can make cutover decisions from one source of truth, as required by the issue.

References

Screenshots or screen recordings

N/A - backend-only change (logging and metrics).

How to set up and validate locally

  1. Enable the feature flag and put an organization in maintenance mode:
    org = Organizations::Organization.find(2)
    org.start_maintenance!(maintenance_reason: 'migration')
    Feature.enable(:organization_maintenance_enforcement, org)
  2. Enqueue a cron worker or org-scoped worker and observe the Sidekiq log for entries with job_status: organization_maintenance_mode.
  3. Check Prometheus metrics for sidekiq_organization_maintenance_mode_jobs_total.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports

Loading
Loading