Rate limit GitLab for Slack app requests per workspace
What does this MR do and why?
Adds a per-Slack-workspace (team_id) rate limit to the four endpoints the GitLab for Slack app calls, enforced after Slack's signature (or the legacy slash-command token) is verified.
Slack traffic reaches GitLab.com from shared Slack IPs with no GitLab credentials, so Rack Attack can only limit it per IP (the Slack carve-out from https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29888 is live). One noisy workspace can still flood these endpoints, and every /events request enqueues a Sidekiq job that may start a Duo Agent Platform session, so that noise hits all of GitLab.com, not just Slack users.
| Key | Endpoints | Limit |
|---|---|---|
slack_app_events |
POST /integrations/slack/events |
60/min per workspace |
slack_app_interactions |
POST /integrations/slack/interactions, POST /integrations/slack/options |
300/min per workspace |
slack_app_commands |
POST /slack/trigger |
60/min per workspace |
Events get a tighter key because they spawn background work; the interactive endpoints are synchronous and bursty (typeahead on /options). Slash commands get their own key so /options chatter can never block /gitlab incident declare during an incident. For scale: Slack itself caps deliveries at 30,000 events per workspace per hour (500/min), and as of September all Slack traffic to GitLab.com combined peaked around 100/hour.
Design points:
- Counted only after verification. Unsigned or wrong-token requests never increment a workspace's counter, so an outsider can't burn a tenant's budget (same reasoning as !260075 (merged) for GitHub webhooks).
- Throttled requests get a 2xx and are dropped, never a 429. Slack retries non-2xx deliveries up to three times (which would amplify a flood) and disables the app's event subscriptions for every workspace once 95% of deliveries fail within an hour. Slash commands return an ephemeral error so the user sees why nothing happened. The signed endpoints also set the
RateLimit-Nameheader on dropped responses. - Log mode first. The limit is always evaluated, so would-be throttles show up in
auth.log(Application_Rate_Limiter_Request; the slash-command entry also carriesslack_workspace) and in labkit'srule_evaluations_total{rule="limit_slack_app_*_by_slack_workspace", result="block"}metric. It is only enforced when theslack_app_workspace_rate_limitfeature flag is enabled. Rollback is disabling the flag. url_verificationevents carry noteam_idand are not counted.
Follow-ups, not in this MR:
- Rollout issue for the flag: watch would-be throttles, then enable on GitLab.com and tune the thresholds.
- A second cap inside
AppMentionedServicebeforetrigger_duo_flow, so DAP session creation per workspace is bounded even when request volume looks normal (raised in https://gitlab.com/gitlab-org/gitlab/-/work_items/631397#note_4011483759).
References
- Resolves https://gitlab.com/gitlab-org/gitlab/-/work_items/631397
- Project Bulkhead, External Agents: https://gitlab.com/groups/gitlab-org/-/work_items/24148
- Rack Attack per-IP carve-out: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29888
- Slack Events API failure limits: https://docs.slack.dev/apis/events-api#failure-limits
How to set up and validate locally
- Set a Slack signing secret and verification token:
Gitlab::CurrentSettings.update!(slack_app_signing_secret: 'secret', slack_app_verification_token: 'token') - Enable the flag:
Feature.enable(:slack_app_workspace_rate_limit) - The slash-command endpoint needs no signature, so it is the easiest to exercise:
The 61st request returns
for i in $(seq 1 60); do curl -s -o /dev/null -w '%{http_code} ' -X POST http://gdk.test:3000/api/v4/slack/trigger \ -d token=token -d text=help -d team_id=T0123ABCDEF done; echo curl -s -X POST http://gdk.test:3000/api/v4/slack/trigger -d token=token -d text=help -d team_id=T0123ABCDEF200with"text":"This endpoint has been requested too many times. Try again later."andlog/auth.loghas anApplication_Rate_Limiter_Requestentry withenv: slack_app_commands_request_limitandslack_workspace: T0123ABCDEF. - Disable the flag and repeat: the request is served normally but the
auth.logentry is still written. - For the signed endpoints, the
slack-simtooling (or any signed request builder) can post anapp_home_openedevent 61 times to/api/v4/integrations/slack/eventswith the sameteam_id; the 61st response is200 {}withRateLimit-Name: slack_app_eventsand noIntegrations::SlackEventWorkerjob enqueued.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.