Add DB-time budgets for GLQL queries
What does this MR do and why?
Adds DB-time budgets for GLQL queries, so one query text, user or guest IP can't keep a large share of the main or CI database busy.
- Cost-mode rate limits, each counting main and CI database seconds (primaries and replicas) in 15-minute windows:
- per query text (
glql_db_seconds_by_query_sha), shared by everyone running that query; - per user (
glql_db_seconds_by_user); a composite identity is charged to the user it acts for; - per user for GitLab Duo Agent Platform flows (
glql_duo_db_seconds_by_user), picked when the token has theai_workflowsscope, which personal access tokens can't carry; - per IP address for unauthenticated requests (
glql_db_seconds_by_ip).
- per query text (
Analytics::Glql::QueryService, used by both/api/glqlandPOST /api/v4/glql, checks the budgets before running a query and charges the request'sdb_main_duration_s,db_main_replica_duration_s,db_ci_duration_sanddb_ci_replica_duration_safter it, including on timeouts.- The limits are application settings (
glql_db_seconds_per_query_limit,glql_db_seconds_per_user_limit,glql_db_seconds_per_ip_limit,glql_duo_db_seconds_per_user_limit), default0, which turns a limit off. Nothing changes until an administrator sets them. - The settings and the GLQL API's budget section are documented as an experiment, since they might be replaced once the cost-based limits are consolidated into a central one, such as !263447.
- Log-only by default: a request over budget logs
GLQL DB-time budget exceededand still runs. With theenforce_glql_db_time_budgetsflag on, it's rejected with the standard rate-limit message, and both endpoints answer 429 withRetry-After(the full 15-minute window). On/api/glqlthe 429 is rendered directly, so it doesn't create an error-tracking event per blocked request. The existing timeout lock keeps its 403 there. - Measure the log-only phase with the
GLQL DB-time budget exceededlog line, not Labkit's metrics. The charge after each request is a counting check, so once a key is over its limit Labkit counts every charge asresult="block"ingitlab_labkit_rate_limiter_rule_evaluations_total, even though nothing is rejected. The checks that do reject are peeks, whichpeeks_totalcounts without a result. - Don't enable the flag before the browser clients handle 429: !267230 (merged)
- Rejections reuse
GlqlQueryLockedError, so they stay out of the GLQL SLIs like other throttled queries.
The labkit rules stay action: :limit. A :log rule over its limit still makes Gitlab::ApplicationRateLimiter return true, so the log-only switch lives in the GLQL code.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/636811
- Rollout issue: #638018
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
-
Set low limits in the Rails console:
ApplicationSetting.current.update!(glql_db_seconds_per_user_limit: 1) -
Run a few GLQL queries as one user, for example through
POST /api/v4/glqlwith a personal access token:curl --request POST "http://gdk.test:3000/api/v4/glql" \ --header "PRIVATE-TOKEN: <token>" \ --data-urlencode 'glql_yaml=query: group = "gitlab-org" AND state = opened' -
Once the user has used more than a second of database time,
log/application_json.logshowsGLQL DB-time budget exceededwithglql_budget_enforced: false, and the query still returns 200. -
Enable enforcement and repeat:
Feature.enable(:enforce_glql_db_time_budgets)Both endpoints now return 429 with a
Retry-Afterheader.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.