Read only the requested customizable dashboard and cache dashboard config per project
Part of Project Bulkhead. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636115
What does this MR do and why?
The legacy customizableDashboards and customizableDashboardVisualizations resolvers read dashboard and visualization YAML from the analytics configuration project through Gitaly: one blob_data_at per dashboard folder, one per visualization file, and one more for every panel that points at a custom visualization. Asking for a single dashboard by slug still loaded every dashboard in the project and filtered the list in Ruby, and nothing was cached, so a repeated request cost exactly as much as the first. The reads land on the Gitaly shard that holds the configuration project.
This MR adds two controls from the issue's list, in two commits:
- Slug lookup reads one folder (unflagged, no behaviour change).
Analytics::Dashboards::Dashboard.find_by_slugchecks the builtin dashboards first and otherwise reads only.gitlab/analytics/dashboards/<slug>/<slug>.yaml. Slugs that cannot name a folder under the dashboards root (visualizations, anything with a path separator,.., a blank) returnnilwithout touching the repository, so the result is identical to filtering the full list.ProductAnalyticsHelpers#product_analytics_dashboardnow calls it. - Per-project config cache behind
customizable_dashboards_config_cache(gitlab_com_derisk, default off).Analytics::Dashboards::ConfigCachekeeps the folder listings and the raw file contents inRails.cache, keyed on[project id, head commit sha, name], so a push to the configuration project invalidates the entries on the next read. A 5 minute TTL is a safety net in case the key ever misses a change (for example a default branch switch that does not move through the paths that expire the repository caches). The cache holds repository content only, never anything derived from the requesting user or the container: authorization stays in the resolvers, and the YAML is still parsed and schema-validated per request, so an invalid file reports its errors the same way with the flag on or off.nilresults are not cached, so a loop over made-up slugs cannot grow the key space, and a file larger thanmax_yaml_size_bytesis returned uncached because the loader rejects it anyway. The dashboards list, the slug lookup, the visualizations list and the per-panel visualization read all share the per-file entries, so a list request warms the slug and panel reads.
Business impact
Customizable dashboards read their YAML from an analytics configuration project through Gitaly: one read per dashboard folder, one per visualization file and one per custom panel, with nothing cached, and a request for a single dashboard by slug still read every dashboard. A Reporter on a group whose configuration project holds 20 dashboards with 3 custom panels each causes roughly 80 Gitaly reads per page load, repeated on every refresh, on a Gitaly node shared with other customers' repositories; the 30-call guard that would stop this in development is not enforced on GitLab.com.
After this MR a slug lookup reads only its own folder (unflagged, same result as before), and with customizable_dashboards_config_cache enabled for the configuration project the folder listings and file contents are served from Rails.cache, keyed on the head commit so a push invalidates them; a warm request makes one FindCommit call and no file reads. Dashboard viewers see the same dashboards, faster; Gitaly on GitLab.com stops paying for repeated reads. No new limit and no new error message. The flag is off by default and its actor is the configuration project (--project=<configuration project>).
Queries this change generates
No PostgreSQL statement is added or changed; db/structure.sql is untouched. The change moves Gitaly RPCs (table above) and adds Redis cache traffic: Rails.cache keys ["analytics_dashboards_config", <config_project_id>, <head_commit_sha>, :dashboard_slugs | :visualization_entries | :blob, <path>] with a 5 minute TTL, blobs read with read_multi and written with write_multi for the misses of one request; nil and oversized files are not cached.
Threshold
Gitaly calls per request, for a configuration project with D dashboards, a requested dashboard with P panels that point at custom visualizations, and V custom visualization files:
| Request | Before | After, flag off | After, flag on and warm |
|---|---|---|---|
| List dashboards | 1 + D | 1 + D | 0 file reads (one FindCommit for the head sha) |
| One dashboard by slug | 1 + D | 1 | 0 file reads |
| Panels of that dashboard | P | P | 0 file reads |
| List visualizations | 1 + V | 1 + V | 0 file reads |
The head commit lookup (FindCommit) stays on every request because it is what invalidates the cache; it is a single cheap call, memoized on the repository instance for the request. config_cache_spec.rb and dashboard_spec.rb assert the warm-cache delta for a freshly loaded project is exactly one Gitaly request. Within one request Gitaly's own 30-call limit is not enforced on GitLab.com, so the slug change alone takes a dashboard page from 1 + D + P to 1 + P reads, and the cache takes a repeated request to zero file reads.
Metering (control 1 in the issue) needs no code: the per-request gitaly_calls field already exists in the Rails logs, so a Kibana query filtered on the customizableDashboards and customizableDashboardVisualizations operation names gives the before and after numbers.
Rollout
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/637957
customizable_dashboards_config_cacheis off by default. Its actor is the configuration project the dashboards are read from, not the group or project viewing them, so enable it with--project=<configuration project>; enabling by group is a no-op. Rollback is turning the flag off; nothing persisted depends on the cache, and the next request reads the repository as before.- The slug-only read (commit 1) ships unflagged. It changes which Gitaly calls are made, not what the resolver returns, and the model specs assert the result against the filtered full list.
Verification
ee/spec/models/analytics/dashboards/dashboard_spec.rb: new.find_by_slugexamples (same result as filtering the full list, reads only the requested folder withexpect(repository).to receive(:blob_data_at).onceandnot_to receive(:tree), fewer Gitaly requests than the list, builtin slugs read nothing, invalid slugs read nothing, empty repository) and cache examples (second request makes no Gitaly request, a new commit to the configuration project is picked up, flag off reads every time, a list request warms the slug lookup, misses are not cached).ee/spec/models/analytics/dashboards/visualization_spec.rb: cache examples for the visualizations list and for.from_file(the per-panel read), and an empty-repository configuration project (Repository#treereturnsnilthere) listing only builtin visualizations without a file read.ee/spec/models/analytics/dashboards/config_cache_spec.rb: key shape and TTL, hit on the second call, miss after the head sha changes,nilnot cached, oversized files returned but not written, duplicate paths read once, warm-cache Gitaly delta of one, empty repository and flag off bypass the cache,blobs_data_atordering.ee/spec/graphql/resolvers/analytics/dashboards/dashboards_resolver_spec.rb: slug lookup does not list the configuration project.ee/spec/requests/api/graphql/analytics/dashboards/dashboards_spec.rb: thereturns all dashboardsexample was quarantined in August under https://gitlab.com/gitlab-org/quality/test-failure-issues/-/issues/24064 (Gitaly call limit hit from a single request) and still expected the product analytics dashboards that have since been removed; it is un-quarantined with the current titles and passes.- Local run against a real Gitaly:
ee/spec/models/analytics/dashboards/,ee/spec/graphql/resolvers/analytics/dashboards/andee/spec/requests/api/graphql/analytics/dashboards/green (213 examples before the review round; the review round re-ran the touched files, 172 examples, andconfig_cache_spec.rb16 examples green on two seeds;visualization_spec.rb64 examples green after the coverage example).
Follow-ups
- Cap the dashboards and panels read per request in the resolvers (control 2 in the issue), once the Kibana numbers show where the cap should sit.
- Decide on
default_enabled: trueand flag removal after the rollout issue is done.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.