Skip to content

Add webhook rate limits for paid plans

Luke Duncalfe requested to merge 337228-webhooks-rate-limited-paid-tier into master

What does this MR do and why?

Customers on the free plan have had their webhooks subject to rate limiting since !61151 (merged).

This change adds the first step towards rate-limiting webhooks for customers on paid plans. In this first step, we just log when a paid plan customer's webhook would be rate-limited.

Free plan customer's webhooks continue to be rate-limited as they are now, except we have changed the rate limit from being per-hook to being per-top-level namespace. This change is reflected in the increase of the Free plan rate limit from the current 120 to 500.

The paid plan limits are stepped according to the number of seats (paid users) that a paid plan customer has. This allows the limit to scale as the customer grows, and power users to be accommodated.

Because the limit logic is more complex, Gitlab::WebHooks::RateLimiter has been added to help abstract it. Some of the code changes are due to this refactor.

#337228 (closed)

How to set up and validate locally

Click to show QA steps
  1. Enable the web_hooks_rate_limit_paid feature flag: 
    Feature.enable(:web_hooks_rate_limit_paid)
  2. Temporarily allow your localhost to use paid plans by applying this patch: 
    diff --git a/ee/app/models/ee/namespace.rb b/ee/app/models/ee/namespace.rb
index 3f223f16b50..77ce40f700e 100644
--- a/ee/app/models/ee/namespace.rb
+++ b/ee/app/models/ee/namespace.rb
@@ -194,7 +194,7 @@ def feature_available_non_trial?(feature)
    override :actual_plan
    def actual_plan
      strong_memoize(:actual_plan) do
-        next ::Plan.default unless ::Gitlab.com?
+        # next ::Plan.default unless ::Gitlab.com?

        if parent_id
          root_ancestor.actual_plan
  3. Choose a top-level group with a project.
  4. Add a "issues events" hook for the group:
    1. For the group, go Settings > Webhooks
    2. For URL, you can generate a unique webhook URL on https://webhook.site.
    3. Check "issues events"
    4. Click Save.
  5. Add a "issues events" hook for the project in the group
    1. For the project, go Settings > Webhooks
    2. For URL, you can generate a unique webhook URL on https://webhook.site.
    3. Check "issues events"
    4. Click Save.
  6. Generate a new paid plan for your top-level group: 
    group = Group.find_by_full_path(<full_path>)

group.root_ancestor.create_gitlab_subscription(
  plan_code: Plan::PREMIUM,
  trial: false,
  start_date: Time.now,
  seats: 1
)

# All going well, this should be true:
group.reload.actual_plan.paid? # => true
  7. Set a low rate limit in your plan limits for testing. Your localhost may not have any plan limits persisted yet: 
    limits = group.reload.actual_plan.actual_limits
old_web_hook_calls_low = limits.web_hook_calls_low # => Keep this to revert back later
limits.save
limits.update!(web_hook_calls_low: 3)
  8. In a terminal window auth.log: tail -f log/auth.log.
  9. Trigger a hook by editing an issue in the project. Do this rapidly and hooks will start to hit the rate limit and logs will appear in log/auth.log. Within a minute their count will reset and executions will not log until the threshold is reached again.
  10. Try these things:
    1. While the project hook is blocked, any hooks of projects within the same top-level group should be blocked also.
    2. While the project hook is blocked, any hook of projects not in the top-level group should not be blocked.
    3. For speed, you can try conducting further tests on the console: 
         Gitlab::WebHooks::RateLimiter.new(hook).rate_limit!   # should always return false, but when over the limit logs should appear
   Gitlab::WebHooks::RateLimiter.new(hook).rate_limited? # should always return false
  11. Clean up:
    1. Undo the patch you applied.
    2. Reset the data: 
      group = Group.find_by_full_path(<full_path>)

limits = group.actual_plan.actual_limits
limits.update!(web_hook_calls_low: old_web_hook_calls_low)

group.root_ancestor.gitlab_subscription.destroy!

Database migration

Up

main: == 20220523030804 AddWebHookCallsMedAndMaxToPlanLimits: migrating =============
main: -- add_column(:plan_limits, :web_hook_calls_mid, :integer, {:null=>false, :default=>0})
main:    -> 0.0027s
main: -- add_column(:plan_limits, :web_hook_calls_low, :integer, {:null=>false, :default=>0})
main:    -> 0.0008s
main: == 20220523030804 AddWebHookCallsMedAndMaxToPlanLimits: migrated (0.0042s) ====

main: == 20220523030805 AddWebHookCallsToPlanLimitsPaidTiers: migrating =============
main: == 20220523030805 AddWebHookCallsToPlanLimitsPaidTiers: migrated (0.0001s) ====

Down

bundle exec rake db:migrate:down:main VERSION=20220523030805

main: == 20220523030805 AddWebHookCallsToPlanLimitsPaidTiers: reverting =============
main: == 20220523030805 AddWebHookCallsToPlanLimitsPaidTiers: reverted (0.0007s) ====

bundle exec rake db:migrate:down:main VERSION=20220523030804

main: == 20220523030804 AddWebHookCallsMedAndMaxToPlanLimits: reverting =============
main: -- remove_column(:plan_limits, :web_hook_calls_low, :integer, {:null=>false, :default=>0})
main:    -> 0.0021s
main: -- remove_column(:plan_limits, :web_hook_calls_mid, :integer, {:null=>false, :default=>0})
main:    -> 0.0003s
main: == 20220523030804 AddWebHookCallsMedAndMaxToPlanLimits: reverted (0.0044s) ====

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #337228 (closed)

Edited by Luke Duncalfe

Merge request reports