Cap custom dashboards and versions per namespace and throttle config updates

Part of Project Bulkhead. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636114

What does this MR do and why?

Custom dashboard writes have no upper bound today: nothing caps how many dashboards a namespace can hold or how many version rows one dashboard accumulates, and no rate limit sits on the config update path. This MR adds three limits behind a new custom_dashboard_write_limits feature flag (gitlab_com_derisk, disabled by default). The flag nests inside custom_dashboard_storage, which still gates the mutations themselves.

1. Dashboards per namespace. Analytics::CustomDashboards::CreateService refuses a new dashboard once its scope already holds MAX_DASHBOARDS_PER_NAMESPACE (50). The scope is the organization plus namespace pair; organization-level dashboards (no namespace) form their own scope. One bounded count runs before the insert (statements as Rails issues them, see the database review section below). It reads the leading columns of the existing idx_unique_dashboard_name_org_namespace (organization_id, namespace_id, name) index. The service returns a ServiceResponse.error ("The limit of 50 custom dashboards for this organization or namespace has been reached") that the createCustomDashboard mutation surfaces in errors.

2. Versions per dashboard. After Analytics::CustomDashboards::UpdateService saves a config change (the only change that writes a custom_dashboard_versions row), it prunes the oldest versions beyond MAX_VERSIONS_PER_DASHBOARD (20), at most VERSION_PRUNE_BATCH_SIZE (100) rows per update, so a dashboard far over the cap converges over a few updates instead of one large delete. One lookup of the newest version number, and, only when that number is past the cap, one bounded delete. Both statements use the existing unique index index_dashboard_versions_on_dashboard_and_version (custom_dashboard_id, version_number), so no new index is needed. The new DashboardVersion.oldest_up_to_version(number) scope keeps the where/order in the model.

   The prune runs after the update and its version row have committed, so it is rescued: a failure (for example a statement timeout on the delete) is reported to error tracking with the dashboard id and the request still returns success. Without that, the client would see an internal error for a change that was saved, and a retry would spend another rate limit slot and write another version.

3. Write rate limit. The createCustomDashboard and updateCustomDashboard mutations share a new custom_dashboard_write application rate limit, 30 requests each minute for each user, registered as a Labkit::RateLimit::Rule in the EE SupportedRateLimits registry and checked through a small WriteRateLimitable concern. Create is always counted; update is counted whenever the request carries config (whether or not it changes), because that is the request shape that can write a version row. Name and description updates are not counted. Throttling create as well closes the delete-and-create loop that would otherwise write 64 KB rows at the general API rate while staying under the dashboard cap. A throttled call returns the standard "requested too many times" message in the mutation errors and is logged to the auth log through throttled_request?, like the other GraphQL rate limits. The fixed limit is documented in doc/rate_limits/non_configurable.md.

With the flag disabled, all three paths are skipped and behaviour is unchanged.

Business impact

Custom dashboards have no write bound today: nothing caps how many dashboards a namespace holds, every config change stores a version row of up to 64 KB that is never removed, and no rate limit sits on the create or update mutations. Any group Developer can update any dashboard in that group, so one person with a script can create dashboards past the 100 the UI lists and push 64 KB rows into the shared custom_dashboard_versions table as fast as the general API throttle allows, growing the table and the WAL for every customer while the read cap hides the extra rows from the UI.

With custom_dashboard_write_limits enabled (off by default, nested inside custom_dashboard_storage): a scope holds at most 50 dashboards, a dashboard keeps its 20 newest versions, and a user may create or change dashboard config 30 times a minute. A user at the dashboard cap sees "The limit of 50 custom dashboards for this organization or namespace has been reached"; a throttled user sees "This endpoint has been requested too many times. Try again later." in the mutation errors; pruning is invisible and never fails a saved update. One namespace scope is now bounded to roughly 67 MB of dashboard content instead of unbounded growth, and the limits exist before custom_dashboard_storage is enabled more widely. Someone editing a dashboard in the UI a few times a minute never reaches any of the limits.

Survivable threshold

• 50 dashboards per namespace scope. The loader's read bound MAX_DB_DASHBOARDS is 100, so a single scope at the cap is still listed in full and the cap never hides rows that the UI would otherwise show. At the 64 KB schema limit per config, a full scope holds at most 3.2 MB of dashboard config.
• 20 versions per dashboard. Enough history for the planned restore use case while bounding one dashboard to 20 x 64 KB = 1.28 MB of version rows. Combined with the dashboard cap, one namespace scope tops out near 50 x 21 x 64 KB, about 67 MB, where today it is unbounded.
• 30 writes each minute for each user, shared between create and config update. A person editing a dashboard in the UI saves a few times a minute; a script at the limit writes at most 30 x 64 KB a minute, and with the two caps that becomes churn inside bounded rows rather than table growth.

The dashboard count is not serialised against concurrent creates, so two requests racing at the cap can both succeed. The cap is approximate by the concurrency, which is acceptable for a protect control; a unique or advisory lock is not worth the cost here.

Database review

Raw SQL as logged from the specs (log/test.log), with literal ids replaced by placeholders. Both tables hold almost no rows on GitLab.com (the feature is behind custom_dashboard_storage), so a Database Lab clone returns 0 rows for every statement. Following the new-feature carve-out, the plans below are EXPLAIN (ANALYZE, BUFFERS) from a local database seeded past the caps inside a rolled-back transaction: 2,120 dashboards across 101 namespaces of one organization and 212,030 version rows, with 130 versions on the dashboard being pruned. Timings are warm-cache and local, so the evidence is the scan type and index, not the milliseconds. Database Lab plans on a seeded clone can be added on request.

1. Dashboard cap, namespace scope (CreateService)

SELECT COUNT(*) FROM (SELECT 1 AS one FROM "custom_dashboards" WHERE "custom_dashboards"."namespace_id" = <namespace_id> AND "custom_dashboards"."organization_id" = <organization_id> LIMIT 50) subquery_for_count

Plan: https://explain.depesz.com/s/SAGm (Index Only Scan on idx_unique_dashboard_name_org_namespace, 50 rows, 0.1 ms, buffers: 3 hit)

2. Dashboard cap, organization scope (CreateService)

SELECT COUNT(*) FROM (SELECT 1 AS one FROM "custom_dashboards" WHERE "custom_dashboards"."namespace_id" IS NULL AND "custom_dashboards"."organization_id" = <organization_id> LIMIT 50) subquery_for_count

Plan: https://explain.depesz.com/s/YfmN (Index Only Scan on idx_unique_dashboard_name_org_namespace, NULLS NOT DISTINCT so the IS NULL branch is an index condition, 50 rows, 0.02 ms, buffers: 13 hit)

3. Newest version number (UpdateService#prune_versions, after a config change)

SELECT MAX("custom_dashboard_versions"."version_number") FROM "custom_dashboard_versions" WHERE "custom_dashboard_versions"."custom_dashboard_id" = <custom_dashboard_id>

Plan: https://explain.depesz.com/s/1MvO (Index Only Scan Backward on index_dashboard_versions_on_dashboard_and_version with LIMIT 1, 1 row, 0.08 ms, buffers: 4 hit)

4. Version prune (UpdateService#prune_versions, only when the newest version number is above 20)

DELETE FROM "custom_dashboard_versions" WHERE ("custom_dashboard_versions"."id") IN (SELECT "custom_dashboard_versions"."id" FROM "custom_dashboard_versions" WHERE "custom_dashboard_versions"."custom_dashboard_id" = <custom_dashboard_id> AND "custom_dashboard_versions"."version_number" <= <newest_version_number - 20> ORDER BY "custom_dashboard_versions"."version_number" ASC LIMIT 100)

Plan: https://explain.depesz.com/s/S0h7 (subquery is an Index Scan on index_dashboard_versions_on_dashboard_and_version, 100 rows; the delete joins back through custom_dashboard_versions_pkey in a Nested Loop, 100 rows deleted, 0.2 ms, buffers: 505 hit including the index and heap writes)

Rollout

• Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/637953
• The flag is disabled by default. Enable it for the same actors that have custom_dashboard_storage enabled, then by default.
• Rollback is turning the flag off; nothing is migrated and no data is changed on rollback. Versions pruned while the flag was on are not restored.
• Enabling the flag on a dashboard that already holds more than 20 versions prunes up to 100 of the oldest on each subsequent config update.

Verification

Run locally against the worktree's own test database, all green after merging master (request specs and registry specs: 111 examples, 0 failures, 3 pending; service and model specs: 41 examples, 0 failures; the pending examples are the pre-existing granular-token shared examples that skip when the namespace has no top-level group):

Spec	Examples	What it covers
ee/spec/services/analytics/custom_dashboards/create_service_spec.rb	10	cap reached refuses the create with no new row; the organization scope and a namespace scope count separately; flag off creates
ee/spec/services/analytics/custom_dashboards/update_service_spec.rb	17	prune keeps the newest versions, removes the oldest first, respects the batch size, skips name-only updates; an update that lands exactly on the cap, or within it, issues no DELETE; a failed prune is tracked and the update still succeeds; flag off keeps every version
ee/spec/requests/api/graphql/mutations/analytics/custom_dashboards/create_spec.rb	20 (plus shared examples)	the 31st write within the window returns the throttled message in errors and creates no row; a call with no request in the GraphQL context (non-HTTP entry point) uses the plain throttle check and is refused the same way; another user is unaffected; flag off creates; a caller without the permission gets the authorization error and the limiter is not consulted
ee/spec/requests/api/graphql/mutations/analytics/custom_dashboards/update_spec.rb	16 (plus shared examples)	the 30th write within the window succeeds and the 31st returns the throttled message in errors without reaching the service; name-only updates are not counted; the throttled call is logged to the auth log; another user is unaffected; flag off updates
ee/spec/lib/ee/gitlab/application_rate_limiter/labkit_adapter/supported_rate_limits_spec.rb	16	the new key is registered
ee/spec/lib/ee/gitlab/application_rate_limiter_spec.rb	32	threshold 30 each minute
ee/spec/models/analytics/custom_dashboards/dashboard_version_spec.rb	12 (plus shared examples)	unchanged model behaviour with the new scope

RuboCop on every changed Ruby file: no offenses. The SQL in the database review section is copied from the statements the specs issued.

Follow-ups

• Mark https://gitlab.com/gitlab-org/gitlab/-/work_items/636114 as a blocker on the custom_dashboard_storage rollout (#582077) so the limits ship before any wider enablement.
• Database Lab plans for the statements in the database review section, before the database review.
• Move the two caps into plan_limits once the flag is on by default; the constants are the cheapest first step while the feature is behind a flag. Follow-up: #638286 (closed).
• Verify the limits are enforced on GitLab.com after enablement, and replay the write pattern from the issue against staging.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.