Add the settings UI for per-project Git rate limits

What does this MR do?

Stores per-project Git rate limit overrides and gives a project Owner a settings UI to set them. Nothing enforces them yet -- that is !256707, which targets this branch.

GitLab's Git HTTP and Git-LFS rate limits are instance-wide. The threshold lives in one ApplicationSetting row and the counter is keyed only by the actor: IP when unauthenticated, user or token when authenticated. So the safe floor for a limit is set by whichever legitimate actor is noisiest anywhere on the instance, and you cannot clamp down one abusive project without taking headroom from every other project.

This MR adds:

  • Three nullable integer columns on project_settings: throttle_unauthenticated_git_http_requests_per_period, throttle_authenticated_git_http_requests_per_period and throttle_authenticated_git_lfs_requests_per_period, with ProjectSetting validations.
  • A new update_git_rate_limit permission, granted to Owner.
  • A settings section under Project > Settings > Repository.
  • Docs for the setting land with the enforcement in !256707, since they describe clamping that only exists there.

Only the request count is overridable. The period still comes from the instance settings. A blank field means no override, which is the state of every project today. When the override is applied, the value is clamped to the instance-wide limit, so an Owner cannot widen a limit an instance admin set. That clamping lives in !256707.

The project_git_throttle_overrides feature flag is enforced in ProjectPolicy as a prevent rule, not re-checked at each call site. can? alone is enough in both the controller and the view, and no caller can check the permission and forget the flag. The flag is disabled by default.

Heads up: git_throttle_overrides_by_full_path, git_throttle_override_redirects, the two track_* methods and the partial index that backs them have no production caller in this MR. Only spec/models/project_setting_spec.rb references them. The caller arrives in !256707. This is deliberate -- storage and UI first, enforcement wiring second -- but I want to say so up front so it is not a surprise.

The two loader methods each have a ceiling (MAX_PROJECTS_WITH_GIT_THROTTLE_OVERRIDE and MAX_GIT_THROTTLE_OVERRIDE_REDIRECTS, both 10_000). Each fetches ceiling+1 rows, truncates to the ceiling, and reports to Gitlab::ErrorTracking when it hit the limit.

Screenshots

Before

image

After

image

image

Database

Two regular migrations: one adds the three nullable integer columns, the other adds the partial index index_project_settings_on_git_throttle_overrides backing the loader query. The index started as a post-deployment migration and moved to a regular one at the database reviewer's request, so the index is in place as soon as the code that needs it deploys.

The loader runs once per process per minute, never on the request path. The plans below are from a Database Lab clone of production.

1. loader join as-is, no project overrides anything
SELECT routes.path, project_settings.project_id,
       project_settings.throttle_unauthenticated_git_http_requests_per_period,
       project_settings.throttle_authenticated_git_http_requests_per_period,
       project_settings.throttle_authenticated_git_lfs_requests_per_period
FROM project_settings
INNER JOIN projects ON projects.id = project_settings.project_id
INNER JOIN routes ON routes.source_type = 'Project' AND routes.source_id = projects.id
WHERE ((project_settings.throttle_unauthenticated_git_http_requests_per_period IS NOT NULL
     OR project_settings.throttle_authenticated_git_http_requests_per_period IS NOT NULL)
     OR project_settings.throttle_authenticated_git_lfs_requests_per_period IS NOT NULL)
ORDER BY project_settings.project_id ASC
LIMIT 10001;
 Limit  (cost=1.26..7.29 rows=1 width=43) (actual time=0.005..0.006 rows=0 loops=1)
   Buffers: shared hit=1
   ->  Nested Loop  (cost=1.26..7.29 rows=1 width=43) (actual time=0.004..0.005 rows=0 loops=1)
         Join Filter: (projects.id = project_settings.project_id)
         Buffers: shared hit=1
         ->  Nested Loop  (cost=0.69..6.68 rows=1 width=47) (actual time=0.004..0.005 rows=0 loops=1)
               Buffers: shared hit=1
               ->  Index Scan using index_project_settings_on_git_throttle_overrides on public.project_settings  (cost=0.12..3.10 rows=1 width=16) (actual time=0.004..0.004 rows=0 loops=1)
                     Buffers: shared hit=1
               ->  Index Scan using index_routes_on_source_type_and_source_id on public.routes  (cost=0.57..3.59 rows=1 width=31) (never executed)
                     Index Cond: (((routes.source_type)::text = 'Project'::text) AND (routes.source_id = project_settings.project_id))
         ->  Index Only Scan using projects_pkey on public.projects  (cost=0.56..0.59 rows=1 width=4) (never executed)
               Index Cond: (projects.id = routes.source_id)
               Heap Fetches: 0
Settings: jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB', work_mem = '230MB'

Time: 4.424 ms (planning 4.365 ms, execution 0.059 ms)
Shared buffers: hits 1 (~8.00 KiB), reads 0

One caveat: this is the plan after ANALYZE project_settings. Before it, with no stats on the new columns, the planner estimated 53M rows and chose a merge join. Production gets that ANALYZE from autovacuum, so it is an artifact of the clone, but worth knowing if anyone re-runs this.

2. same loader at the ceiling, 10,000 seeded overrides + ANALYZE

Seeded with UPDATE project_settings SET throttle_authenticated_git_http_requests_per_period = 600 WHERE project_id IN (SELECT project_id FROM project_settings TABLESAMPLE SYSTEM (0.05) LIMIT 10000), then ANALYZE project_settings. Same query as 1.

 Limit  (cost=34663.08..35541.64 rows=7530 width=43) (actual time=59.482..69.874 rows=10000 loops=1)
   Buffers: shared hit=109256
   ->  Gather Merge  (cost=34663.08..35541.64 rows=7530 width=43) (actual time=59.480..69.106 rows=10000 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=109256
         ->  Sort  (cost=33663.06..33672.47 rows=3765 width=43) (actual time=55.497..55.884 rows=3333 loops=3)
               Sort Key: project_settings.project_id
               Sort Method: quicksort  Memory: 389kB
               Buffers: shared hit=109256
               ->  Nested Loop  (cost=120.61..33439.45 rows=3765 width=43) (actual time=1.498..53.993 rows=3333 loops=3)
                     Buffers: shared hit=109238
                     ->  Nested Loop  (cost=120.05..30349.71 rows=4817 width=20) (actual time=1.460..30.740 rows=3333 loops=3)
                           Buffers: shared hit=59236
                           ->  Parallel Bitmap Heap Scan on public.project_settings  (cost=119.48..20435.18 rows=4817 width=16) (actual time=1.416..17.097 rows=3333 loops=3)
                                 Recheck Cond: ((project_settings.throttle_unauthenticated_git_http_requests_per_period IS NOT NULL) OR (project_settings.throttle_authenticated_git_http_requests_per_period IS NOT NULL) OR (project_settings.throttle_authenticated_git_lfs_requests_per_period IS NOT NULL))
                                 Heap Blocks: exact=3123
                                 Buffers: shared hit=8950
                                 ->  Bitmap Index Scan on index_project_settings_on_git_throttle_overrides  (cost=0.00..116.59 rows=11561 width=0) (actual time=2.661..2.662 rows=10000 loops=1)
                                       Buffers: shared hit=38
                           ->  Index Only Scan using projects_pkey on public.projects  (cost=0.56..2.06 rows=1 width=4) (actual time=0.004..0.004 rows=1 loops=10000)
                                 Index Cond: (projects.id = project_settings.project_id)
                                 Heap Fetches: 412
                                 Buffers: shared hit=50286
                     ->  Index Scan using index_routes_on_source_type_and_source_id on public.routes  (cost=0.57..0.64 rows=1 width=31) (actual time=0.007..0.007 rows=1 loops=10000)
                           Index Cond: (((routes.source_type)::text = 'Project'::text) AND (routes.source_id = projects.id))
                           Buffers: shared hit=50002
Settings: jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB', work_mem = '230MB'
Query ID: -1837963907654307872

Time: 74.954 ms (planning 4.523 ms, execution 70.431 ms)
Shared buffers: hits 109256 (~853.60 MiB), reads 0

Still drives from index_project_settings_on_git_throttle_overrides. The cost is the two 10,000-iteration nested loops hanging off it. Cold on the clone this was 5.9s with 17s of I/O wait.

3. redirect_routes with a 10,000 element source_id list

The ids are synthetic (every 5300th id across the range) -- Joe returns plans, not rows, so the seeded ids could not be read back out. 831 of the 10,000 matched real redirect rows.

SELECT redirect_routes.path, redirect_routes.source_id
FROM redirect_routes
WHERE redirect_routes.source_type = 'Project'
  AND redirect_routes.source_id IN (<10000 ids>)
ORDER BY redirect_routes.id ASC
LIMIT 10001;
 Limit  (cost=43786.03..43811.03 rows=10001 width=44) (actual time=41.761..41.887 rows=831 loops=1)
   Buffers: shared hit=39115
   ->  Sort  (cost=43786.03..43845.24 rows=23686 width=44) (actual time=41.759..41.814 rows=831 loops=1)
         Sort Key: redirect_routes.id
         Sort Method: quicksort  Memory: 100kB
         Buffers: shared hit=39115
         ->  Index Scan using index_redirect_routes_on_source_type_and_source_id on public.redirect_routes  (cost=0.56..42093.92 rows=23686 width=44) (actual time=0.219..41.375 rows=831 loops=1)
               Index Cond: (((redirect_routes.source_type)::text = 'Project'::text) AND (redirect_routes.source_id = ANY ('{...10000 ids...}'::integer[])))
               Buffers: shared hit=39112
Settings: work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB'
Query ID: 4776222978431714012

Time: 91.164 ms (planning 49.167 ms, execution 41.997 ms)
Shared buffers: hits 39115 (~305.60 MiB), reads 0

Two caveats on the numbers. Thin-clone I/O is much slower than production, so the buffer counts transfer but the cold timings don't. And plan 3's ids are synthetic (every 5300th id across the range) because Joe returns plans, not rows, so the seeded ids couldn't be read back out; 831 of the 10,000 matched real redirect rows.

Relevant issues

Related to #629903

Edited by John Cai

Merge request reports

Loading
Loading