Read only the requested customizable dashboard and cache dashboard config per project

Part of Project Bulkhead. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636115

What does this MR do and why?

The legacy customizableDashboards and customizableDashboardVisualizations resolvers read dashboard and visualization YAML from the analytics configuration project through Gitaly: one blob_data_at per dashboard folder, one per visualization file, and one more for every panel that points at a custom visualization. Asking for a single dashboard by slug still loaded every dashboard in the project and filtered the list in Ruby, and nothing was cached, so a repeated request cost exactly as much as the first. The reads land on the Gitaly shard that holds the configuration project.

This MR adds two controls from the issue's list, in two commits:

  1. Slug lookup reads one folder (unflagged, no behaviour change). Analytics::Dashboards::Dashboard.find_by_slug checks the builtin dashboards first and otherwise reads only .gitlab/analytics/dashboards/<slug>/<slug>.yaml. Slugs that cannot name a folder under the dashboards root (visualizations, anything with a path separator, .., a blank) return nil without touching the repository, so the result is identical to filtering the full list. ProductAnalyticsHelpers#product_analytics_dashboard now calls it.
  2. Per-project config cache behind customizable_dashboards_config_cache (gitlab_com_derisk, default off). Analytics::Dashboards::ConfigCache keeps the folder listings and the raw file contents in Rails.cache, keyed on [project id, head commit sha, name], so a push to the configuration project invalidates the entries on the next read. A 5 minute TTL is a safety net in case the key ever misses a change (for example a default branch switch that does not move through the paths that expire the repository caches). The cache holds repository content only, never anything derived from the requesting user or the container: authorization stays in the resolvers, and the YAML is still parsed and schema-validated per request, so an invalid file reports its errors the same way with the flag on or off. nil results are not cached, so a loop over made-up slugs cannot grow the key space, and a file larger than max_yaml_size_bytes is returned uncached because the loader rejects it anyway. The dashboards list, the slug lookup, the visualizations list and the per-panel visualization read all share the per-file entries, so a list request warms the slug and panel reads.

Business impact

Customizable dashboards read their YAML from an analytics configuration project through Gitaly: one read per dashboard folder, one per visualization file and one per custom panel, with nothing cached, and a request for a single dashboard by slug still read every dashboard. A Reporter on a group whose configuration project holds 20 dashboards with 3 custom panels each causes roughly 80 Gitaly reads per page load, repeated on every refresh, on a Gitaly node shared with other customers' repositories; the 30-call guard that would stop this in development is not enforced on GitLab.com.

After this MR a slug lookup reads only its own folder (unflagged, same result as before), and with customizable_dashboards_config_cache enabled for the configuration project the folder listings and file contents are served from Rails.cache, keyed on the head commit so a push invalidates them; a warm request makes one FindCommit call and no file reads. Dashboard viewers see the same dashboards, faster; Gitaly on GitLab.com stops paying for repeated reads. No new limit and no new error message. The flag is off by default and its actor is the configuration project (--project=<configuration project>).

Queries this change generates

No PostgreSQL statement is added or changed; db/structure.sql is untouched. The change moves Gitaly RPCs (table above) and adds Redis cache traffic: Rails.cache keys ["analytics_dashboards_config", <config_project_id>, <head_commit_sha>, :dashboard_slugs | :visualization_entries | :blob, <path>] with a 5 minute TTL, blobs read with read_multi and written with write_multi for the misses of one request; nil and oversized files are not cached.

Threshold

Gitaly calls per request, for a configuration project with D dashboards, a requested dashboard with P panels that point at custom visualizations, and V custom visualization files:

Request Before After, flag off After, flag on and warm
List dashboards 1 + D 1 + D 0 file reads (one FindCommit for the head sha)
One dashboard by slug 1 + D 1 0 file reads
Panels of that dashboard P P 0 file reads
List visualizations 1 + V 1 + V 0 file reads

The head commit lookup (FindCommit) stays on every request because it is what invalidates the cache; it is a single cheap call, memoized on the repository instance for the request. config_cache_spec.rb and dashboard_spec.rb assert the warm-cache delta for a freshly loaded project is exactly one Gitaly request. Within one request Gitaly's own 30-call limit is not enforced on GitLab.com, so the slug change alone takes a dashboard page from 1 + D + P to 1 + P reads, and the cache takes a repeated request to zero file reads.

Metering (control 1 in the issue) needs no code: the per-request gitaly_calls field already exists in the Rails logs, so a Kibana query filtered on the customizableDashboards and customizableDashboardVisualizations operation names gives the before and after numbers.

Rollout

  • Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/637957
  • customizable_dashboards_config_cache is off by default. Its actor is the configuration project the dashboards are read from, not the group or project viewing them, so enable it with --project=<configuration project>; enabling by group is a no-op. Rollback is turning the flag off; nothing persisted depends on the cache, and the next request reads the repository as before.
  • The slug-only read (commit 1) ships unflagged. It changes which Gitaly calls are made, not what the resolver returns, and the model specs assert the result against the filtered full list.

Verification

  • ee/spec/models/analytics/dashboards/dashboard_spec.rb: new .find_by_slug examples (same result as filtering the full list, reads only the requested folder with expect(repository).to receive(:blob_data_at).once and not_to receive(:tree), fewer Gitaly requests than the list, builtin slugs read nothing, invalid slugs read nothing, empty repository) and cache examples (second request makes no Gitaly request, a new commit to the configuration project is picked up, flag off reads every time, a list request warms the slug lookup, misses are not cached).
  • ee/spec/models/analytics/dashboards/visualization_spec.rb: cache examples for the visualizations list and for .from_file (the per-panel read), and an empty-repository configuration project (Repository#tree returns nil there) listing only builtin visualizations without a file read.
  • ee/spec/models/analytics/dashboards/config_cache_spec.rb: key shape and TTL, hit on the second call, miss after the head sha changes, nil not cached, oversized files returned but not written, duplicate paths read once, warm-cache Gitaly delta of one, empty repository and flag off bypass the cache, blobs_data_at ordering.
  • ee/spec/graphql/resolvers/analytics/dashboards/dashboards_resolver_spec.rb: slug lookup does not list the configuration project.
  • ee/spec/requests/api/graphql/analytics/dashboards/dashboards_spec.rb: the returns all dashboards example was quarantined in August under https://gitlab.com/gitlab-org/quality/test-failure-issues/-/issues/24064 (Gitaly call limit hit from a single request) and still expected the product analytics dashboards that have since been removed; it is un-quarantined with the current titles and passes.
  • Local run against a real Gitaly: ee/spec/models/analytics/dashboards/, ee/spec/graphql/resolvers/analytics/dashboards/ and ee/spec/requests/api/graphql/analytics/dashboards/ green (213 examples before the review round; the review round re-ran the touched files, 172 examples, and config_cache_spec.rb 16 examples green on two seeds; visualization_spec.rb 64 examples green after the coverage example).

Follow-ups

  • Cap the dashboards and panels read per request in the resolvers (control 2 in the issue), once the Kibana numbers show where the cap should sit.
  • Decide on default_enabled: true and flag removal after the rollout issue is done.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chandra Saripaka

Merge request reports

Loading
Loading