Bound group activity counts and cache timeouts
What does this MR do and why?
The group overview page calls three group activity endpoints: GET /api/v4/analytics/group_activity/{merge_requests,issues,new_members}_count. On a very large group, a count can run until the 15 second statement timeout. The timeout raises inside Rails.cache.fetch, so nothing is cached, and the next page view runs the same query again.
This MR bounds each count and remembers the failure:
- Each count runs on a replica with a 4.5 second statement timeout.
- When the count is cancelled, the calculator caches a "too large" result for 1 hour under the same cache key, and returns it without running the query again.
- The API returns
nullfor the count and a newtoo_large: truefield. The field is only present when the count timed out, so normal responses do not change. - The group activity widget shows an em dash with a "This count took too long to calculate" tooltip instead of an error alert.
The behavior is behind the group_activity_fast_count_timeout feature flag (gitlab_com_derisk, group actor, default off). With the flag off, counts behave as before.
Why not ApplicationRecord.with_fast_read_statement_timeout. That helper opens a transaction for SET LOCAL statement_timeout and relies on fallback_to_replicas_for_ambiguous_queries. That fallback is skipped when the load balancing session sticks to the primary (for example after a recent write by the same user). The transaction and SET LOCAL then run on the primary, while the count still reads from a replica under use_replicas_for_read_queries, so the short timeout never applies. This MR opens the transaction on the replica connection through load_balancer.read. The count runs on the same connection, because the load balancer caches the host for the request. A :database_replica spec checks both cases: a fresh session, and a session stuck to the primary. With the helper swapped in, the sticky case fails with SET LOCAL on the primary.
No changelog entry: the change is behind a default-off flag. The API doc page will get a note about too_large when the flag is enabled by default. The generated OpenAPI file is updated.
Business impact
Every group overview page calls three group activity count endpoints, open to any member of a Premium or Ultimate group. On a very large group a count can run until the 15 second statement timeout; the timeout is raised inside Rails.cache.fetch, so nothing is cached, the page shows "Failed to load group activity metrics", and the next page view runs the same 15 second query on a shared replica again, once per member because the cache key includes the user id.
With group_activity_fast_count_timeout enabled for the group, each count runs on a replica with a 4.5 second timeout; a cancelled count is remembered for 1 hour under the same key and returned as a null count with "too_large": true, so a very large group costs one bounded query per member per hour instead of an unbounded one per page view. The widget shows an em dash with the tooltip "This count took too long to calculate" instead of an error. Groups that count in time see no change: too_large is absent from their responses. The flag is off by default; with it off, behaviour is unchanged.
Queries this change generates
No new query. Each existing count is wrapped, on the replica connection, in (captured from ee/spec/lib/analytics/group_activity_calculator_spec.rb, log/test.log, literals replaced):
BEGIN
SET LOCAL statement_timeout = 4500
SELECT COUNT(*) FROM (SELECT 1 AS one FROM "merge_requests" ... AND "merge_requests"."created_at" >= '<now - 30 days>' LIMIT 1000) subquery_for_count
COMMITThe issues and members counts are wrapped identically; the inner statements are the unchanged IssuesFinder, MergeRequestsFinder and GroupMembersFinder queries. Predicates and indexes are unchanged (index_merge_requests_on_target_project_id_and_created_at_and_id, idx_issues_on_project_id_and_created_at_and_id_and_state_id, index_members_on_source_and_type_and_access_level); the SET LOCAL only bounds their runtime. On QueryCanceled the calculator writes too_large to the Redis key ["groups", "recent_30d_<type>_count", <group_id>, <user_id>] for 1 hour (derived from code).
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636162
- Feature flag rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/637996
Screenshots or screen recordings
The widget when the merge request count times out, rendered from the real component with the GitLab stylesheets and sample counts, not production data. The merge requests tile shows an em dash; its tooltip reads "This count took too long to calculate".
| Before | After |
|---|---|
| Error alert: "Failed to load group activity metrics" | em dash with tooltip "This count took too long to calculate" |
How to set up and validate locally
- In the Rails console, enable the flag for a Premium or Ultimate group:
Feature.enable(:group_activity_fast_count_timeout, Group.find_by_full_path('your-group')) - Force a timeout: temporarily set
STATEMENT_TIMEOUT_MS = 1inee/lib/analytics/group_activity_calculator.rband restart Rails (gdk restart rails-web). A 1 ms timeout cancels most counts; a very small group may still finish in time. - Clear the cached counts:
Rails.cache.delete_matched('*recent_30d_*'). - Call
GET /api/v4/analytics/group_activity/merge_requests_count?group_path=your-group. The response is{"merge_requests_count":null,"too_large":true}. - Visit the group overview page. The widget shows an em dash with the tooltip "This count took too long to calculate".
- Disable the flag and clear the cache. The response is a plain count again. Revert the constant when done.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
