[FF] skip_noop_integration_jobs -- Skip enqueueing no-op integration jobs

Summary

Roll out Bound Integrations::ExecuteWorker fan-out currently behind the skip_noop_integration_jobs feature flag.

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

This flag covers both MR A (chat/email no-op skipping) and MR C (Jira no-op skipping). The risk is low because:

  • Chat/email integrations: notify_for_pipeline? already returns false for most pipeline statuses (only failed and success notify). We're just moving the check to enqueue time.
  • Jira: Integrations::Jira#execute is a no-op. Cross-references are created by Integrations::CreateExternalCrossReferenceWorker, not by ExecuteWorker.

Blast radius: If the pre-check logic is wrong, some legitimate notifications could be skipped. Rollback is immediate via disabling the flag.

Dashboards to watch:

  • gitlab_integrations_enqueue_skipped_total by integration_class and event
  • Integrations::ExecuteWorker done jobs in Kibana split by json.extra.integrations_execute_worker.integration_class.keyword
  • catchall-b shard share for the worker (~12.9% over 7 days before)
  • Integrations::CreateExternalCrossReferenceWorker volume (should stay flat, as Jira cross-references don't go through ExecuteWorker)

Events

Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging

Rollout

Run all production chatops in #production and cross-post the results to #g_import. Background: incremental rollout process, feature actors.

Non-production

chatops gitlab run feature set skip_noop_integration_jobs true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

chatops gitlab run feature set skip_noop_integration_jobs 1 --actors
chatops gitlab run feature set skip_noop_integration_jobs 10 --actors
chatops gitlab run feature set skip_noop_integration_jobs 50 --actors
chatops gitlab run feature set skip_noop_integration_jobs 100 --actors

The actor here is the project or group (not the root namespace like the budget flag), so a percentage rollout splits big namespaces, which is fine for this flag.

What to watch at each step

  • gitlab_integrations_enqueue_skipped_total by integration_class and event — should grow with each step, mostly pipeline for chat/email and merge_request for Jira
  • Integrations::ExecuteWorker done jobs in Kibana split by json.extra.integrations_execute_worker.integration_class.keyword:
    • Baseline: ~20.4M/day
    • Chat/email types: ~22% of jobs
    • Jira: ~7.6% (1.54M/day)
    • Expected: Jira should go close to zero, chat/email should drop but not to zero (failed/success pipelines still notify). At most ~30% less jobs in total. Datadog should stay the same.
  • catchall-b shard share for the worker (~12.9% over 7 days before) and its Sidekiq seconds
  • Integrations::CreateExternalCrossReferenceWorker volume stays flat (Jira cross-references don't go through ExecuteWorker)
  • Quick check on a test project that a failed pipeline still posts to Slack, and that notify_only_broken_pipelines still behaves as before

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Post-rollout

Once at 100% for a couple of days (including a weekday), re-measure the per-namespace data for the integration_execution_budget flag rollout (#637439), as the numbers used for the budget include the no-op jobs.

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

chatops gitlab run release check <merge-request-url> <milestone>
chatops gitlab run feature delete skip_noop_integration_jobs --dev --pre --staging --staging-ref --production

Rollback

chatops gitlab run feature set skip_noop_integration_jobs false                                         # production
chatops gitlab run feature set skip_noop_integration_jobs false --dev --pre --staging --staging-ref     # non-production
chatops gitlab run feature delete skip_noop_integration_jobs --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Carla Drago