Rate limit GitLab for Slack app requests per workspace

What does this MR do and why?

Adds a per-Slack-workspace (team_id) rate limit to the four endpoints the GitLab for Slack app calls, enforced after Slack's signature (or the legacy slash-command token) is verified.

Slack traffic reaches GitLab.com from shared Slack IPs with no GitLab credentials, so Rack Attack can only limit it per IP (the Slack carve-out from https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29888 is live). One noisy workspace can still flood these endpoints, and every /events request enqueues a Sidekiq job that may start a Duo Agent Platform session, so that noise hits all of GitLab.com, not just Slack users.

Key Endpoints Limit
slack_app_events POST /integrations/slack/events 60/min per workspace
slack_app_interactions POST /integrations/slack/interactions, POST /integrations/slack/options 300/min per workspace
slack_app_commands POST /slack/trigger 60/min per workspace

Events get a tighter key because they spawn background work; the interactive endpoints are synchronous and bursty (typeahead on /options). Slash commands get their own key so /options chatter can never block /gitlab incident declare during an incident. For scale: Slack itself caps deliveries at 30,000 events per workspace per hour (500/min), and as of September all Slack traffic to GitLab.com combined peaked around 100/hour.

Design points:

  • Counted only after verification. Unsigned or wrong-token requests never increment a workspace's counter, so an outsider can't burn a tenant's budget (same reasoning as !260075 (merged) for GitHub webhooks).
  • Throttled requests get a 2xx and are dropped, never a 429. Slack retries non-2xx deliveries up to three times (which would amplify a flood) and disables the app's event subscriptions for every workspace once 95% of deliveries fail within an hour. Slash commands return an ephemeral error so the user sees why nothing happened. The signed endpoints also set the RateLimit-Name header on dropped responses.
  • Log mode first. The limit is always evaluated, so would-be throttles show up in auth.log (Application_Rate_Limiter_Request; the slash-command entry also carries slack_workspace) and in labkit's rule_evaluations_total{rule="limit_slack_app_*_by_slack_workspace", result="block"} metric. It is only enforced when the slack_app_workspace_rate_limit feature flag is enabled. Rollback is disabling the flag.
  • url_verification events carry no team_id and are not counted.

Follow-ups, not in this MR:

References

How to set up and validate locally

  1. Set a Slack signing secret and verification token:
    Gitlab::CurrentSettings.update!(slack_app_signing_secret: 'secret', slack_app_verification_token: 'token')
  2. Enable the flag:
    Feature.enable(:slack_app_workspace_rate_limit)
  3. The slash-command endpoint needs no signature, so it is the easiest to exercise:
    for i in $(seq 1 60); do
      curl -s -o /dev/null -w '%{http_code} ' -X POST http://gdk.test:3000/api/v4/slack/trigger \
        -d token=token -d text=help -d team_id=T0123ABCDEF
    done; echo
    curl -s -X POST http://gdk.test:3000/api/v4/slack/trigger -d token=token -d text=help -d team_id=T0123ABCDEF
    The 61st request returns 200 with "text":"This endpoint has been requested too many times. Try again later." and log/auth.log has an Application_Rate_Limiter_Request entry with env: slack_app_commands_request_limit and slack_workspace: T0123ABCDEF.
  4. Disable the flag and repeat: the request is served normally but the auth.log entry is still written.
  5. For the signed endpoints, the slack-sim tooling (or any signed request builder) can post an app_home_opened event 61 times to /api/v4/integrations/slack/events with the same team_id; the 61st response is 200 {} with RateLimit-Name: slack_app_events and no Integrations::SlackEventWorker job enqueued.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jannik Lehmann

Merge request reports

Loading
Loading