Draft: [skip ci] Log clients over a main primary database held-time budget
Stacked MR. This targets
feature/accurate-primary-session-time(!263117) so the diff shows only this change. Retarget tomasterafter !263117 merges.
What does this MR do and why?
Logs web clients that hold the main primary database for more than 600 session-seconds a minute (10 sessions held continuously). A client is the signed-in user, or the IP when signed out. This is log-only and behind the default-off log_primary_database_time_per_client flag.
The budget is hard-coded for now (600 s per 60 s, THRESHOLD and INTERVAL in the middleware). That's deliberate while it's log-only: the first job is to measure the normal ceiling on GitLab.com. It moves to an application setting before it enforces anything, so it can be tuned without an MR. That's tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/634464 and noted in the code.
In INC-15305, one project bot held over 100 of the main primary's roughly 144 database sessions. It stayed under every request-count limit, and nothing measured how much primary time a single client used.
- A new cost-mode rate limit,
main_db_held_duration_limit_per_client, with characteristics%i[user ip]:{ user: id }when signed in, otherwise{ ip: }. This matchescheck_rate_limit_by_user_or_ip!and the other*_by_user_or_iprules. - A Rack middleware,
Gitlab::Middleware::PrimaryDatabaseTimePerClient, runs at the end of each web request. If the request used the main primary, it charges the request'sdb_main_held_duration_sto that client's budget and logs the client if it's over budget.
It never rejects a request. Enforcement would be a later, separate change, made once the normal ceiling is measured on GitLab.com.
Design notes and trade-offs
-
What's charged:
db_main_held_duration_sfrom !263117, read throughHeldTime.payload. It's outermost transaction time plus primary statements run outside any transaction, so it's the time the request held a primary session and doesn't double count savepoints. Held session-seconds per minute ÷ 60 is the average number of sessions the client held. -
Threshold: 600 s per 60 s equals 10 sessions held continuously. Replayed against INC-15305's logs, the bot passed it at 15:56–15:57 and reached at least 108 sessions' worth (6,514 s in the 15:59 minute, a lower bound from whole-second logs). The normal ceiling isn't measured yet; that's what the log-only phase is for.
-
Where the setting will live: probably
resource_usage_limits, next to Sidekiq's per-worker budgets of the same kind. Sidekiq currently treats any rule in that setting as a full replacement for its defaults, so it has to ignore rules it doesn't own first. That's a separate change, made in parallel. -
Same mechanism as Sidekiq: the cost mode of
Gitlab::ApplicationRateLimiter, as used bymain_db_duration_limit_per_worker. The middleware passes the held time ascost:tothrottled?, becauseresource_usage_throttled?reads its cost from aSafeRequestStorekey and held time isn't stored under one. -
Middleware position: inserted directly after
Gitlab::Middleware::RequestContext, which is itself insideRequestStore::Middleware. After the app returns:- the request store still holds the held-time stats; it's cleared only when
RequestStore::Middlewareexits. - the user and IP pushed by controllers and the API (
Gitlab::ApplicationContext.push) are still on the context stack; they're popped only whenRequestContext'swith_contextexits.
A spec asserts this ordering, and I checked it end to end with request specs through the full stack (API signed in and anonymous, and a controller signed in).
- the request store still holds the held-time stats; it's cleared only when
-
User and IP lookup:
Gitlab::ApplicationContext.current_context_attribute(Labkit::Fields::GL_USER_ID), elsecurrent_context_attribute(:remote_ip). Requests with neither (no controller or API context) are skipped. -
Requests that didn't use the main primary skip Redis. Those that did pay one rate-limiter Redis call before the response is returned. About 26,000 requests a minute open a main-primary transaction at weekday peak, so that's at least about 440 calls a second; requests with only statements outside a transaction add to that.
-
Not counted: requests killed before they finish (they never run
ensure); Sidekiq, which already has per-worker budgets; and time a request is still in flight. This measures usage over the minute, not instantaneous concurrency. -
Known gaps:
- Shared IPs: signed-out users behind one NAT or proxy, CI runners, and internal callers without a user (for example KAS or gitlab-shell calls not tied to a key's user) share one IP budget. The log line carries
meta.caller_id, so these can be filtered out. - Spreading across identities: an actor using many accounts or IPs stays under each budget.
- Capacity loss isn't per client: when the primary slows or fails over, every client's held time rises at once; no single client is at fault and this budget doesn't help.
- Log-only: nothing is rejected yet.
- Shared IPs: signed-out users behind one NAT or proxy, CI runners, and internal callers without a user (for example KAS or gitlab-shell calls not tied to a key's user) share one IP budget. The log line carries
-
Log volume: a client over budget logs once per request while over, bounded by that client's own request rate and only during an event.
-
Instance-wide flag: a percentage rollout would charge only part of each client's usage.
-
Log values: sub-second values in Kibana need gitlab-com/gl-infra/observability/elastic-cloud!284 (merged) (durations mapped as
double). The budget is charged in Rails and doesn't depend on it.
References
- Tracking and rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/634464
- Incident INC-15305: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23261
- Depends on: !263117
- Log mapping: gitlab-com/gl-infra/observability/elastic-cloud!284 (merged)
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
- In the Rails console, enable the flag:
Feature.enable(:log_primary_database_time_per_client) - To see a log line locally, temporarily set
THRESHOLD = 0.001inlib/gitlab/middleware/primary_database_time_per_client.rband restart Rails. - Signed in, perform a write, for example post a comment on an issue.
- In
log/application_json.log, find"message":"Client is over its main primary held time budget"withgl_user_idset to your user ID anddb_main_held_duration_sset to the request's held time. - Signed out, sign in with a wrong password for an existing user, which writes
failed_attemptsto the main primary. The line hasremote_ipinstead ofgl_user_id. - Revert the
THRESHOLDchange afterwards.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
- Changelog: none (default-off flag).
- Feature flag:
log_primary_database_time_per_client(gitlab_com_derisk, default off, actor:instance).