Add per-namespace execution budget for integrations

What does this MR do and why?

Part of https://gitlab.com/gitlab-org/gitlab/-/work_items/636614, following !266040 (merged).

Integrations::ExecuteWorker runs about 15M jobs a day, and a few namespaces can account for a large share of the load on one Sidekiq shard. This MR adds a per top-level namespace, per minute budget on enqueues, checked in async_execute right after the skip_enqueue? check from !266040 (merged). Events that are skipped as no-ops never spend budget.

Merging this MR changes no behavior. Both limits default to 0 (unlimited) and all feature flags are off.

How it works

  • Budget key per class. Integration#execution_budget_key returns :integration_executions by default.
    • Integrations::Datadog returns :integration_executions_datadog. Datadog is about 68% of jobs, so it gets its own bucket and doesn't consume the budget for chat and other integrations.
    • Integrations::Jira returns nil (exempt). Its #execute is a no-op, so it shouldn't spend budget.
  • Limits. Two new plan_limits columns, integration_executions and integration_executions_datadog (integer, default 0, null: false). 0 means unlimited.
  • Rate limiting. Two new rules in the labkit rate limit registry, each with its own counter. They are keyed by root namespace, with a 1 minute period and the threshold passed by the caller.
  • Integrations::ExecutionBudget. Resolves the root namespace from the project or group and reads the plan limit.
    • Integrations with no parent (instance, template, organization) are never throttled.
    • The namespace and limit lookups are memoized per instance (strong_memoize_attr) and in Gitlab::SafeRequestStore, because one event fans out to several integrations of the same project.
  • async_execute. over_execution_budget? resolves the root namespace of the project or group and uses it as the feature flag actor. It returns false immediately when integration_execution_budget is off, so there are no plan limit lookups or counter increments on that path.
    • When throttled, it increments gitlab_integrations_execution_budget_exceeded_total with the labels integration_class, event, bucket and mode (enforced or dry_run). There is no namespace label, to keep cardinality down.
    • It logs at most once per root namespace and bucket per minute.
    • In dry-run it still enqueues the job. When enforced, it drops the job.
    • It fails open. Any error, for example from Redis, is tracked with Gitlab::ErrorTracking.track_exception and the job is enqueued as before.
  • API and docs. Both limits are exposed in the admin plan limits API, with docs in doc/api/plan_limits.md, doc/administration/instance_limits.md and doc/development/webhooks.md. The limits are documented as beta, so they can be renamed or removed without the deprecation process.

Feature flags

Flag Purpose
integration_execution_budget Master switch. Counts and logs would-be throttles (dry-run). Actor is the root namespace.
enforce_integration_execution_budget Drops over-budget enqueues for the shared bucket. Actor is the root namespace.
enforce_integration_execution_budget_datadog Drops over-budget enqueues for the Datadog bucket. Actor is the root namespace.

All three are gitlab_com_derisk and default off. There is one enforce flag per bucket so the shared budget can be enforced while Datadog stays in dry-run.

The flags use the root namespace as the actor, the same as the budget counter and Gitlab::WebHooks::RateLimiter. A percentage-of-actors rollout therefore covers whole namespaces: either every project and group in a root namespace is counted and throttled, or none of them are. With a project or group actor, only some projects of a namespace would increment its counter. That would undercount the per-namespace peaks used to pick enforce values, and projects in the same namespace would be throttled differently.

Rollout plan

  1. Merge with both limits at 0 and all flags off.
  2. Confirm skip_noop_integration_jobs (from !266040 (merged)) is enabled on GitLab.com, ideally near 100%. The shared bucket's per-namespace peaks should be re-measured after that, because the existing numbers include no-op chat jobs.
  3. On GitLab.com, set provisional dry-run values on the free plan, as for the web hooks limit:
    • Shared bucket: about 500/min.
    • Datadog bucket: about 3,000/min as a placeholder until Datadog-only per-namespace data is available.
  4. Enable integration_execution_budget for a small percentage of actors. Actors are root namespaces, so the percentage is a share of namespaces, and each selected namespace is counted in full. Use the counter and logs to see which namespaces trip each limit.
  5. Set enforce values above the observed peaks. Because whole namespaces are counted, the observed peaks are complete per-namespace figures. Get sign-off from the integrations owners (#599597) before enabling any enforce flag. Enforce the shared bucket first and Datadog last. Enable the enforce flags for the same share of namespaces, or a smaller one, than the master flag.

Rollback: disable the flags, or set the limit back to 0.

Risks and notes

  • Dropped events are not retried when enforced, because ExecuteWorker uses dead: false. For Datadog, a dropped pipeline or job event may mean a partial trace. Keep its limit generous and enforce it last.
  • Retries can double-count. For example, a retried Ci::ExecuteBuildHooksWorker can count twice. This is a throttle, not a ledger.
  • The 1 minute window is approximate. Protection is uneven at window edges. This is acceptable for protecting the shard.
  • This bounds the worst-case namespace, not the total. Cutting the aggregate volume would need CI hook batching or a concurrency_limit, which were parked in refinement.
  • Jira no-op jobs. Jira is exempt from the budget here. Skipping its jobs entirely is handled in !267033 (merged).

Database

Adds two integer columns with a constant default of 0 and null: false to plan_limits.

How to set up and validate locally

bundle exec rspec \
  spec/lib/integrations/execution_budget_spec.rb \
  spec/lib/gitlab/application_rate_limiter/labkit_adapter/supported_rate_limits_spec.rb \
  spec/models/plan_limits_spec.rb \
  spec/models/integrations/datadog_spec.rb \
  spec/models/integration_spec.rb \
  spec/requests/api/admin/plan_limits_spec.rb

To try it in a console:

Plan.default.actual_limits.update!(integration_executions: 5)
Feature.enable(:integration_execution_budget)
# Dry-run: jobs are still enqueued and the counter increments.
Feature.enable(:enforce_integration_execution_budget)
# Enforced: over-budget enqueues are dropped.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #636614

Edited by Carla Drago

Merge request reports

Loading
Loading