Skip enqueueing no-op integration jobs
What does this MR do and why?
Integrations::ExecuteWorker runs about 15M jobs a day (12.9% of catchall-b). Part of that volume is jobs that exit without doing anything.
A pipeline fires hooks on every status transition. Every chat and pipeline email integration gets a job for each one. But notify_for_pipeline? returns false for every status except failed and success, and for success when notify_only_broken_pipelines is on (the default for new chat integrations). In those cases #execute sends nothing, and the job only used Sidekiq and Redis capacity.
This MR skips those jobs at enqueue time:
- Adds
Integration#should_enqueue?(data), which defaults totrue.async_executecalls it after the existingsupported_eventscheck. ChatNotificationoverrides it. It returnsfalseunlessshould_execute?passes (supported event and webhook present). Forpipelineevents it also requiresnotify_for_pipeline?(data).PipelinesEmailoverrides it forpipelineevents withnotify_for_pipeline?(data)only.- Skips are counted in
gitlab_integrations_enqueue_skipped_total, labeled byintegration_classandevent. There is no log line, because the volume would be very high. - The behavior is behind the
skip_noop_integration_jobsfeature flag (gitlab_com_derisk, default off).
Part of https://gitlab.com/gitlab-org/gitlab/-/work_items/636614 (MR A of the plan to bound Integrations::ExecuteWorker fan-out).
Design notes
- No Gitaly at enqueue. Neither override calls
notify_for_ref?or the branch and source checks, because they can callrepository.expand_ref. Those checks stay in the worker. - Worker-side checks are unchanged. Jobs already in the queue, and any event the pre-check lets through, behave as before. The pre-check mirrors the status logic that
#executealready applies, so it can only skip jobs that would have done nothing. - Data shape.
async_executealready converts the payload to indifferent access, and it is the same payload the worker receives. - Flag off.
should_enqueue?is still evaluated, but the result is ignored and the job is enqueued as before. The check only reads the payload and the integration's own properties. - Follow-up. MR B (per-namespace budget) will call its limiter after this check, so skipped no-op events won't spend budget.
How to set up and validate locally
bundle exec rspec \
spec/models/integration_spec.rb \
spec/models/integrations/base/chat_notification_spec.rb \
spec/models/integrations/pipelines_email_spec.rb \
spec/workers/integrations/execute_worker_spec.rbTo see it manually, in a Rails console with an active Slack integration (default settings) on a project:
Feature.enable(:skip_noop_integration_jobs)
integration = project.integrations.find_by(type: 'Integrations::Slack')
# Not enqueued, counted as skipped
integration.async_execute(object_kind: 'pipeline', object_attributes: { status: 'running' })
# Enqueued
integration.async_execute(object_kind: 'pipeline', object_attributes: { status: 'failed' })Tests:
#should_enqueue?defaults totrue.- A skip enqueues nothing and increments the counter when the flag is on. It still enqueues, with no count, when the flag is off.
- Chat: a status table (
failed,success,running,pending,created,canceled,skipped) againstnotify_only_broken_pipelinesand status-change-only mode. It also covers unsupported events and a blank webhook, and assertsnotify_for_ref?is not called. - Pipelines email: a status table, a non-pipeline event, and a check that the branch and source checks are not called.
- Regression: with default Slack settings, a
runningpipeline event enqueues no job and afailedone does.
Rollout
- Merge with the flag off. There is no behavior change.
- Enable
skip_noop_integration_jobson GitLab.com, starting with a small percentage of actors. - Compare
gitlab_integrations_enqueue_skipped_totalwithIntegrations::ExecuteWorkerjob volume. This gives the real share of no-op jobs. That share is not measured yet, so I'm not claiming a percentage here. - Remove the flag once the numbers look right.
Rollback: disable the flag. Jobs are enqueued as before, and nothing is persisted.
Risk
- Low. The only user-visible effect is that a notification which would not have been sent is no longer queued.
- A wrong override could suppress a real notification. The status table specs and the unchanged worker-side logic cover that. Enable the flag on a small percentage first.
- Not covered: Datadog subscribes to every pipeline and build event and can't be pre-checked this way, so it is unaffected.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #636614