Requeue Sidekiq jobs for organizations in maintenance mode

What does this MR do and why?

Sidekiq jobs belonging to an organization in maintenance mode were being dropped. This MR buffers them in Redis instead and re-enqueues them once the organization exits maintenance, so no work is lost. It was split out of MR !252853 (closed) at the reviewer's suggestion — that MR (now Draft) only added observability for the drop.

Design notes

The solution introduces five new components:

• Gitlab::SidekiqMiddleware::OrganizationMaintenanceMode::QueueManager — per-organization Redis list holding buffered jobs, plus a Redis set indexing which organizations have buffers. Batches of up to 1000, guarded by an exclusive lease, 30-day TTL refreshed on write.
• Gitlab::SidekiqMiddleware::OrganizationMaintenanceMode::Server — server middleware that buffers instead of running the job.
• Gitlab::SidekiqMiddleware::OrganizationMaintenanceMode::Resume — client middleware that replays the buffered metadata onto each job as it is re-enqueued.
• Gitlab::SidekiqLogging::OrganizationMaintenanceModeLogger — structured log per intercepted job; actions are buffered, skipped, filtered, resumed. Uses the standard Labkit::Fields::GL_ORGANIZATION_ID field. Also increments the sidekiq_organization_maintenance_mode_jobs_total Prometheus counter with labels worker and action.
• Organizations::MaintenanceMode::ResumeWorker — cron job running every minute; drains only organizations that are no longer under_maintenance?.

Behaviour by state:

• Steady maintenance? state: buffer the job, do not run it.
• maintenance_initialization? (the drain window): run the job and log it as filtered, so outstanding work can finish.
• Cron workers: skipped, not buffered, because they reschedule themselves and buffering would replay every missed tick at once.
• Organization deleted while jobs sit buffered: the buffer is dropped rather than re-enqueued.

Design decisions:

1. Middleware placement. OrganizationMaintenanceMode::Server is registered before DuplicateJobs::Server. A job intercepted after deduplication has already had its cookie deleted; later duplicates would not deduplicate and the buffer would fill with copies. Intercepting earlier keeps the cookie alive, mirroring the existing ConcurrencyLimit::Server placement.
2. The deduplication exemption. Because the cookie stays live, the re-enqueued job would be deduplicated against itself. A new organization_maintenance_mode_resume flag exempts it, mirroring concurrency_limit_resume.
3. Reading the organization. The middleware reads organization_id from Gitlab::ApplicationContext rather than Current.organization, because CurrentOrganization::Server runs at the end of the server chain and would not have populated it yet.
4. Metadata replay. bulk_perform_async cannot set per-job fields, so QueueManager stores one metadata entry per job and the Resume client middleware pops them in enqueue order. This preserves jid, application context, wal_locations, and compression markers — without them a resumed job loses log correlation and a data_consistency :delayed worker could read a stale replica.

Known trade-offs

• The per-job database lookup is gated on a cached list from the Organizations::Stateful.in_maintenance_states scope, so no query runs for a job whose organization is in no maintenance state. Every enforcement decision is still resolved against the real record, which makes Rails.cache only a gate. The remaining trade-off is that a state transition can lag by up to the 30 second TTL, which is acceptable for an operator-driven window measured in hours or days.
• Undrainable buffered jobs (unparseable payload, or a worker class that no longer exists) are trimmed rather than retried forever. Parse failures are reported to error tracking.
• No new feature flag: the behaviour is already gated by under_maintenance?, which requires the existing default-off organization_maintenance_enforcement flag.

Database review

New scope in Organizations::Stateful:

MAINTENANCE_STATES = %i[maintenance_initialization maintenance]

scope :in_maintenance_states, -> { where(state: MAINTENANCE_STATES) }

maintenance_initialization and maintenance map to state enum values 5 and 6.

Caller: Gitlab::SidekiqMiddleware::OrganizationMaintenanceMode::Server#candidate_organization_ids. It calls .pluck_primary_key on this scope inside Rails.cache.fetch with a 30 second TTL, so the query runs at most once per TTL per cache key, not once per job.

Generated SQL:

SELECT "organizations"."id" FROM "organizations" WHERE "organizations"."state" IN (5, 6)

Execution plan (local GDK, organizations has only 2 rows — not representative of production volume):

Index Scan using index_organizations_on_state on organizations  (cost=0.15..6.25 rows=6 width=8) (actual time=0.005..0.005 rows=0 loops=1)
  Index Cond: (state = ANY ('{5,6}'::integer[]))
  Buffers: shared hit=3 read=1
Planning:
  Buffers: shared hit=159 read=8
Planning Time: 2.835 ms
Execution Time: 0.008 ms

Uses the existing index_organizations_on_state btree index. No new index added.

No LIMIT, by design: this query backs maintenance-mode enforcement, not a paginated listing. Truncating the result would silently stop enforcing maintenance mode for whichever organizations fell outside the page — a correctness bug, not a performance concern. The result set is bounded by operator action instead: it only ever contains organizations an operator has explicitly put into a maintenance state, expected to be a handful at most.

Net effect on load: this scope replaces a per-job query. The middleware previously ran Organizations::Organization.find_by_id (a primary-key lookup) for every Sidekiq job carrying an organization_id, instance-wide, even with the feature flag off everywhere. This change reduces that to one cached, index-scoped query per 30 seconds.

Requesting database review to confirm the plan holds at production volume via Database Lab, since the plan above is from a 2-row local table.

Testing

249 examples pass across the new and touched specs, covering the queue manager against real Redis (FIFO order, TTL, batching, multi-worker grouping, lease contention, undrainable payloads), the server middleware per state, metadata replay, the resume worker, the logger, the deduplication exemption, the cached candidate gate (no lookup when the organization is in no maintenance state, one refresh across repeated jobs, a stale cached id whose row is gone), and the in_maintenance_states scope. RuboCop is clean on all 18 Ruby files of the 20 changed.

References

Closes #628636 (closed)

• Sibling work item (observability): #602826 (closed)
• Epic: gitlab-org#20404
• Split from: !252853 (closed)