[FF] skip_noop_integration_jobs -- Skip enqueueing no-op integration jobs
Summary
Roll out Bound Integrations::ExecuteWorker fan-out currently behind the skip_noop_integration_jobs feature flag.
- DRI: @carlad-gl
- Team Slack channel:
#g_import - cc @stefanosxan
Note
Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle
What could go wrong?
This flag covers both MR A (chat/email no-op skipping) and MR C (Jira no-op skipping). The risk is low because:
- Chat/email integrations:
notify_for_pipeline?already returns false for most pipeline statuses (onlyfailedandsuccessnotify). We're just moving the check to enqueue time. - Jira:
Integrations::Jira#executeis a no-op. Cross-references are created byIntegrations::CreateExternalCrossReferenceWorker, not byExecuteWorker.
Blast radius: If the pre-check logic is wrong, some legitimate notifications could be skipped. Rollback is immediate via disabling the flag.
Dashboards to watch:
gitlab_integrations_enqueue_skipped_totalbyintegration_classandeventIntegrations::ExecuteWorkerdone jobs in Kibana split byjson.extra.integrations_execute_worker.integration_class.keyword- catchall-b shard share for the worker (~12.9% over 7 days before)
Integrations::CreateExternalCrossReferenceWorkervolume (should stay flat, as Jira cross-references don't go throughExecuteWorker)
Events
- Exceptions with skip_noop_integration_jobs:1
- Events with skip_noop_integration_jobs:1
- Error rate and other graphs by modifying the examples in the Visualization Library
Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging
Rollout
Run all production chatops in #production and cross-post the results to #g_import. Background: incremental rollout process, feature actors.
Non-production
chatops gitlab run feature set skip_noop_integration_jobs true --dev --pre --staging --staging-refProduction — percentage rollout (wait ≥15 min between steps, watch dashboards):
chatops gitlab run feature set skip_noop_integration_jobs 1 --actors
chatops gitlab run feature set skip_noop_integration_jobs 10 --actors
chatops gitlab run feature set skip_noop_integration_jobs 50 --actors
chatops gitlab run feature set skip_noop_integration_jobs 100 --actorsThe actor here is the project or group (not the root namespace like the budget flag), so a percentage rollout splits big namespaces, which is fine for this flag.
What to watch at each step
gitlab_integrations_enqueue_skipped_totalbyintegration_classandevent— should grow with each step, mostlypipelinefor chat/email andmerge_requestfor JiraIntegrations::ExecuteWorkerdone jobs in Kibana split byjson.extra.integrations_execute_worker.integration_class.keyword:- Baseline: ~20.4M/day
- Chat/email types: ~22% of jobs
- Jira: ~7.6% (1.54M/day)
- Expected: Jira should go close to zero, chat/email should drop but not to zero (failed/success pipelines still notify). At most ~30% less jobs in total. Datadog should stay the same.
- catchall-b shard share for the worker (~12.9% over 7 days before) and its Sidekiq seconds
Integrations::CreateExternalCrossReferenceWorkervolume stays flat (Jira cross-references don't go throughExecuteWorker)- Quick check on a test project that a failed pipeline still posts to Slack, and that
notify_only_broken_pipelinesstill behaves as before
Before global rollout
Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:
- Docs + version history updated
- Breaking changes announced, if any
- Change management issue opened, if required
- External API consumers handled with a fail-open mechanism, if applicable
Post-rollout
Once at 100% for a couple of days (including a weekday), re-measure the per-namespace data for the integration_execution_budget flag rollout (#637439), as the numbers used for the budget include the no-op jobs.
Cleanup
Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:
chatops gitlab run release check <merge-request-url> <milestone>
chatops gitlab run feature delete skip_noop_integration_jobs --dev --pre --staging --staging-ref --productionRollback
chatops gitlab run feature set skip_noop_integration_jobs false # production
chatops gitlab run feature set skip_noop_integration_jobs false --dev --pre --staging --staging-ref # non-production
chatops gitlab run feature delete skip_noop_integration_jobs --dev --pre --staging --staging-ref --production # remove entirely