Skip enqueueing no-op integration jobs

What does this MR do and why?

Integrations::ExecuteWorker runs about 15M jobs a day (12.9% of catchall-b). Part of that volume is jobs that exit without doing anything.

A pipeline fires hooks on every status transition. Every chat and pipeline email integration gets a job for each one. But notify_for_pipeline? returns false for every status except failed and success, and for success when notify_only_broken_pipelines is on (the default for new chat integrations). In those cases #execute sends nothing, and the job only used Sidekiq and Redis capacity.

This MR skips those jobs at enqueue time:

  • Adds Integration#should_enqueue?(data), which defaults to true. async_execute calls it after the existing supported_events check.
  • ChatNotification overrides it. It returns false unless should_execute? passes (supported event and webhook present). For pipeline events it also requires notify_for_pipeline?(data).
  • PipelinesEmail overrides it for pipeline events with notify_for_pipeline?(data) only.
  • Skips are counted in gitlab_integrations_enqueue_skipped_total, labeled by integration_class and event. There is no log line, because the volume would be very high.
  • The behavior is behind the skip_noop_integration_jobs feature flag (gitlab_com_derisk, default off).

Part of https://gitlab.com/gitlab-org/gitlab/-/work_items/636614 (MR A of the plan to bound Integrations::ExecuteWorker fan-out).

Design notes

  • No Gitaly at enqueue. Neither override calls notify_for_ref? or the branch and source checks, because they can call repository.expand_ref. Those checks stay in the worker.
  • Worker-side checks are unchanged. Jobs already in the queue, and any event the pre-check lets through, behave as before. The pre-check mirrors the status logic that #execute already applies, so it can only skip jobs that would have done nothing.
  • Data shape. async_execute already converts the payload to indifferent access, and it is the same payload the worker receives.
  • Flag off. should_enqueue? is still evaluated, but the result is ignored and the job is enqueued as before. The check only reads the payload and the integration's own properties.
  • Follow-up. MR B (per-namespace budget) will call its limiter after this check, so skipped no-op events won't spend budget.

How to set up and validate locally

bundle exec rspec \
  spec/models/integration_spec.rb \
  spec/models/integrations/base/chat_notification_spec.rb \
  spec/models/integrations/pipelines_email_spec.rb \
  spec/workers/integrations/execute_worker_spec.rb

To see it manually, in a Rails console with an active Slack integration (default settings) on a project:

Feature.enable(:skip_noop_integration_jobs)
integration = project.integrations.find_by(type: 'Integrations::Slack')

# Not enqueued, counted as skipped
integration.async_execute(object_kind: 'pipeline', object_attributes: { status: 'running' })

# Enqueued
integration.async_execute(object_kind: 'pipeline', object_attributes: { status: 'failed' })

Tests:

  • #should_enqueue? defaults to true.
  • A skip enqueues nothing and increments the counter when the flag is on. It still enqueues, with no count, when the flag is off.
  • Chat: a status table (failed, success, running, pending, created, canceled, skipped) against notify_only_broken_pipelines and status-change-only mode. It also covers unsupported events and a blank webhook, and asserts notify_for_ref? is not called.
  • Pipelines email: a status table, a non-pipeline event, and a check that the branch and source checks are not called.
  • Regression: with default Slack settings, a running pipeline event enqueues no job and a failed one does.

Rollout

  1. Merge with the flag off. There is no behavior change.
  2. Enable skip_noop_integration_jobs on GitLab.com, starting with a small percentage of actors.
  3. Compare gitlab_integrations_enqueue_skipped_total with Integrations::ExecuteWorker job volume. This gives the real share of no-op jobs. That share is not measured yet, so I'm not claiming a percentage here.
  4. Remove the flag once the numbers look right.

Rollback: disable the flag. Jobs are enqueued as before, and nothing is persisted.

Risk

  • Low. The only user-visible effect is that a notification which would not have been sent is no longer queued.
  • A wrong override could suppress a real notification. The status table specs and the unchanged worker-side logic cover that. Enable the flag on a small percentage first.
  • Not covered: Datadog subscribes to every pipeline and build event and can't be pre-checked this way, so it is unaffected.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #636614

Merge request reports

Loading
Loading