Draft: Add Sidekiq observability for organization maintenance mode skip/cancel/filter events
What does this MR do and why?
Implements structured logging and Prometheus metrics for every Sidekiq job that is skipped, cancelled, or filtered because its organization is in maintenance mode. This is part of #602826 (closed) (Org Read-Only Mode: observability for skip/cancel/filter events), which is part of the epic gitlab-org#20404 (Implement Organization Maintenance Mode).
New components
Gitlab::SidekiqLogging::OrganizationMaintenanceModeLogger (singleton, follows PauseControlLogger/DeduplicationLogger patterns):
- Emits a structured log entry to the Sidekiq logger for each intercepted job
- Payload fields:
organization_maintenance_mode.action(skipped|cancelled|filtered),organization_maintenance_mode.organization_id,class,jid,job_status,pid
Gitlab::SidekiqMiddleware::OrganizationMaintenanceMode::Server (server-side middleware):
- Placed after
CurrentOrganization::ServersoCurrent.organizationis already set - Cron workers (
queue_namespace == 'cronjob') in maintenance -> skip (raisesSidekiq::JobRetry::Skip, not re-enqueued) - Org-scoped workers with
maintenance_enforced?-> cancel (dropped without retry) - Org-scoped workers in maintenance but without enforcement -> filter (logged, job proceeds)
- Each action increments
sidekiq_organization_maintenance_mode_jobs_totalPrometheus counter (labels:worker,action)
Design rationale
The single log/metric signal is shared by:
- Steady-state observability dashboards and alerts
- The cutover-readiness endpoint (drain completion check)
This means operators can make cutover decisions from one source of truth, as required by the issue.
References
- Work item: #602826 (closed)
- Epic: gitlab-org#20404
- ADR 010: https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/organization/decisions/010_organization_maintenance_mode/
- Block-all enforcement MR: !250225 (merged)
- Feature flag:
organization_maintenance_enforcement
Screenshots or screen recordings
N/A - backend-only change (logging and metrics).
How to set up and validate locally
- Enable the feature flag and put an organization in maintenance mode:
org = Organizations::Organization.find(2) org.start_maintenance!(maintenance_reason: 'migration') Feature.enable(:organization_maintenance_enforcement, org) - Enqueue a cron worker or org-scoped worker and observe the Sidekiq log for entries with
job_status: organization_maintenance_mode. - Check Prometheus metrics for
sidekiq_organization_maintenance_mode_jobs_total.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.