Add DB-time budgets for GLQL queries

What does this MR do and why?

Adds DB-time budgets for GLQL queries, so one query text, user or guest IP can't keep a large share of the main or CI database busy.

  • Cost-mode rate limits, each counting main and CI database seconds (primaries and replicas) in 15-minute windows:
    • per query text (glql_db_seconds_by_query_sha), shared by everyone running that query;
    • per user (glql_db_seconds_by_user); a composite identity is charged to the user it acts for;
    • per user for GitLab Duo Agent Platform flows (glql_duo_db_seconds_by_user), picked when the token has the ai_workflows scope, which personal access tokens can't carry;
    • per IP address for unauthenticated requests (glql_db_seconds_by_ip).
  • Analytics::Glql::QueryService, used by both /api/glql and POST /api/v4/glql, checks the budgets before running a query and charges the request's db_main_duration_s, db_main_replica_duration_s, db_ci_duration_s and db_ci_replica_duration_s after it, including on timeouts.
  • The limits are application settings (glql_db_seconds_per_query_limit, glql_db_seconds_per_user_limit, glql_db_seconds_per_ip_limit, glql_duo_db_seconds_per_user_limit), default 0, which turns a limit off. Nothing changes until an administrator sets them.
  • The settings and the GLQL API's budget section are documented as an experiment, since they might be replaced once the cost-based limits are consolidated into a central one, such as !263447.
  • Log-only by default: a request over budget logs GLQL DB-time budget exceeded and still runs. With the enforce_glql_db_time_budgets flag on, it's rejected with the standard rate-limit message, and both endpoints answer 429 with Retry-After (the full 15-minute window). On /api/glql the 429 is rendered directly, so it doesn't create an error-tracking event per blocked request. The existing timeout lock keeps its 403 there.
  • Measure the log-only phase with the GLQL DB-time budget exceeded log line, not Labkit's metrics. The charge after each request is a counting check, so once a key is over its limit Labkit counts every charge as result="block" in gitlab_labkit_rate_limiter_rule_evaluations_total, even though nothing is rejected. The checks that do reject are peeks, which peeks_total counts without a result.
  • Don't enable the flag before the browser clients handle 429: !267230 (merged)
  • Rejections reuse GlqlQueryLockedError, so they stay out of the GLQL SLIs like other throttled queries.

The labkit rules stay action: :limit. A :log rule over its limit still makes Gitlab::ApplicationRateLimiter return true, so the log-only switch lives in the GLQL code.

References

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. Set low limits in the Rails console:

    ApplicationSetting.current.update!(glql_db_seconds_per_user_limit: 1)
  2. Run a few GLQL queries as one user, for example through POST /api/v4/glql with a personal access token:

    curl --request POST "http://gdk.test:3000/api/v4/glql" \
      --header "PRIVATE-TOKEN: <token>" \
      --data-urlencode 'glql_yaml=query: group = "gitlab-org" AND state = opened'
  3. Once the user has used more than a second of database time, log/application_json.log shows GLQL DB-time budget exceeded with glql_budget_enforced: false, and the query still returns 200.

  4. Enable enforcement and repeat:

    Feature.enable(:enforce_glql_db_time_budgets)

    Both endpoints now return 429 with a Retry-After header.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Daniele Rossetti

Merge request reports

Loading
Loading