## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/630507) currently behind the `glql_dashboard_panels_in_reading_order` feature flag. - DRI: @blabuschagne - Team Slack channel: `#g_analytics_optimize` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? - The flag is `gitlab_com_derisk`, introduced by https://gitlab.com/gitlab-org/gitlab/-/merge_requests/258689. With it on, every GLQL panel on an analytics dashboard view queues its query on mount, in reading order. Without it, panels more than half a viewport below the fold wait until the user scrolls near them. - Blast radius: all analytics dashboards with GLQL panels, for users the flag is enabled for. This covers the Explore system dashboards such as DAP Impact, and group and project custom dashboards. - Load: there are more GLQL API requests per view. On the DAP Impact Work tab that is 33 requests per tab switch instead of 17 on switch plus 16 more only when the user scrolls. Requests still go through the per-viewer dashboard queue, 6 in flight at a time (8 if https://gitlab.com/gitlab-org/gitlab/-/merge_requests/258678 merges). Peak concurrency per viewer does not change, but total requests per view do. The GLQL API (`/api/glql`) hits ClickHouse for analytics sources and Postgres for the rest. - There is no data-loss risk. The change only orders and starts client-side requests. The client-only `loadPriority` value is stripped before a dashboard is saved. - If the priority plumbing has a bug, requests would run in a different order and not be dropped. A panel without a priority falls back to arrival order. - Rollback: disabling the flag returns panels to viewport deferral on the next page load. There is no migration or cache to undo. Watch: GLQL API request rate, error rate and latency for `/api/glql`, and ClickHouse query duration and memory, while the percentage rises. ## Events - [Exceptions with glql_dashboard_panels_in_reading_order:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22glql_dashboard_panels_in_reading_order:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with glql_dashboard_panels_in_reading_order:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22glql_dashboard_panels_in_reading_order:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_analytics_optimize`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set glql_dashboard_panels_in_reading_order 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set glql_dashboard_panels_in_reading_order true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set glql_dashboard_panels_in_reading_order <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss glql_dashboard_panels_in_reading_order true /chatops gitlab run feature set --group=gitlab-org,gitlab-com glql_dashboard_panels_in_reading_order true /chatops gitlab run feature set --user=blabuschagne glql_dashboard_panels_in_reading_order true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete glql_dashboard_panels_in_reading_order --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set glql_dashboard_panels_in_reading_order false # production /chatops gitlab run feature set glql_dashboard_panels_in_reading_order false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete glql_dashboard_panels_in_reading_order --dev --pre --staging --staging-ref --production # remove entirely ```