Draft: [skip ci] Log clients over a main primary database held-time budget

Stacked MR. This targets feature/accurate-primary-session-time (!263117) so the diff shows only this change. Retarget to master after !263117 merges.

What does this MR do and why?

Logs web clients that hold the main primary database for more than 600 session-seconds a minute (10 sessions held continuously). A client is the signed-in user, or the IP when signed out. This is log-only and behind the default-off log_primary_database_time_per_client flag.

The budget is hard-coded for now (600 s per 60 s, THRESHOLD and INTERVAL in the middleware). That's deliberate while it's log-only: the first job is to measure the normal ceiling on GitLab.com. It moves to an application setting before it enforces anything, so it can be tuned without an MR. That's tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/634464 and noted in the code.

In INC-15305, one project bot held over 100 of the main primary's roughly 144 database sessions. It stayed under every request-count limit, and nothing measured how much primary time a single client used.

  • A new cost-mode rate limit, main_db_held_duration_limit_per_client, with characteristics %i[user ip]: { user: id } when signed in, otherwise { ip: }. This matches check_rate_limit_by_user_or_ip! and the other *_by_user_or_ip rules.
  • A Rack middleware, Gitlab::Middleware::PrimaryDatabaseTimePerClient, runs at the end of each web request. If the request used the main primary, it charges the request's db_main_held_duration_s to that client's budget and logs the client if it's over budget.

It never rejects a request. Enforcement would be a later, separate change, made once the normal ceiling is measured on GitLab.com.

Design notes and trade-offs
  • What's charged: db_main_held_duration_s from !263117, read through HeldTime.payload. It's outermost transaction time plus primary statements run outside any transaction, so it's the time the request held a primary session and doesn't double count savepoints. Held session-seconds per minute ÷ 60 is the average number of sessions the client held.

  • Threshold: 600 s per 60 s equals 10 sessions held continuously. Replayed against INC-15305's logs, the bot passed it at 15:56–15:57 and reached at least 108 sessions' worth (6,514 s in the 15:59 minute, a lower bound from whole-second logs). The normal ceiling isn't measured yet; that's what the log-only phase is for.

  • Where the setting will live: probably resource_usage_limits, next to Sidekiq's per-worker budgets of the same kind. Sidekiq currently treats any rule in that setting as a full replacement for its defaults, so it has to ignore rules it doesn't own first. That's a separate change, made in parallel.

  • Same mechanism as Sidekiq: the cost mode of Gitlab::ApplicationRateLimiter, as used by main_db_duration_limit_per_worker. The middleware passes the held time as cost: to throttled?, because resource_usage_throttled? reads its cost from a SafeRequestStore key and held time isn't stored under one.

  • Middleware position: inserted directly after Gitlab::Middleware::RequestContext, which is itself inside RequestStore::Middleware. After the app returns:

    • the request store still holds the held-time stats; it's cleared only when RequestStore::Middleware exits.
    • the user and IP pushed by controllers and the API (Gitlab::ApplicationContext.push) are still on the context stack; they're popped only when RequestContext's with_context exits.

    A spec asserts this ordering, and I checked it end to end with request specs through the full stack (API signed in and anonymous, and a controller signed in).

  • User and IP lookup: Gitlab::ApplicationContext.current_context_attribute(Labkit::Fields::GL_USER_ID), else current_context_attribute(:remote_ip). Requests with neither (no controller or API context) are skipped.

  • Requests that didn't use the main primary skip Redis. Those that did pay one rate-limiter Redis call before the response is returned. About 26,000 requests a minute open a main-primary transaction at weekday peak, so that's at least about 440 calls a second; requests with only statements outside a transaction add to that.

  • Not counted: requests killed before they finish (they never run ensure); Sidekiq, which already has per-worker budgets; and time a request is still in flight. This measures usage over the minute, not instantaneous concurrency.

  • Known gaps:

    • Shared IPs: signed-out users behind one NAT or proxy, CI runners, and internal callers without a user (for example KAS or gitlab-shell calls not tied to a key's user) share one IP budget. The log line carries meta.caller_id, so these can be filtered out.
    • Spreading across identities: an actor using many accounts or IPs stays under each budget.
    • Capacity loss isn't per client: when the primary slows or fails over, every client's held time rises at once; no single client is at fault and this budget doesn't help.
    • Log-only: nothing is rejected yet.
  • Log volume: a client over budget logs once per request while over, bounded by that client's own request rate and only during an event.

  • Instance-wide flag: a percentage rollout would charge only part of each client's usage.

  • Log values: sub-second values in Kibana need gitlab-com/gl-infra/observability/elastic-cloud!284 (merged) (durations mapped as double). The budget is charged in Rails and doesn't depend on it.

References

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. In the Rails console, enable the flag:
    Feature.enable(:log_primary_database_time_per_client)
  2. To see a log line locally, temporarily set THRESHOLD = 0.001 in lib/gitlab/middleware/primary_database_time_per_client.rb and restart Rails.
  3. Signed in, perform a write, for example post a comment on an issue.
  4. In log/application_json.log, find "message":"Client is over its main primary held time budget" with gl_user_id set to your user ID and db_main_held_duration_s set to the request's held time.
  5. Signed out, sign in with a wrong password for an existing user, which writes failed_attempts to the main primary. The line has remote_ip instead of gl_user_id.
  6. Revert the THRESHOLD change afterwards.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

  • Changelog: none (default-off flag).
  • Feature flag: log_primary_database_time_per_client (gitlab_com_derisk, default off, actor :instance).
Edited by Alex Pooley

Merge request reports

Loading
Loading