VSM & Group::Optimize Dashboards - adding instance-level / organization-level analytics

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Overview

• Our GL.com instance-level metrics are not visible to customers. These pages/APIs are only for admins. On self-managed, these pages are accessible to instance admins only.
• Self-managed allows you to have many top-level groups.
• The scalability of instance-level analytics is a concern, as it may impact system performance in high-load scenarios when aggregating data across numerous groups, projects or users.

More context here.

Problem to solve

To optimize SDLC, software leaders need a centralized UI to see and compare all of GitLab's instance-level DevSecOps metrics. With PostgreSQL groupoptimize can't scale up to these analytical workloads.

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/353651+ / &10484 (comment 1405482547) / #413202 (comment 1980742928)

Proposal

There is an ongoing project to replicate some of the tables to ClickHouse using a new tool called Siphon.

The MVP initially covers the following tables (no CI tables):

• namespaces
• projects
• events
• issues
• merge_requests
• namespace_details
• bulk_import_entities
• milestones
• notes

Once data replication is stable and working, product teams (Optimize) can start planning/thinking about analytical features (note: the initial rollout is SaaS only). We replicate all table columns so organization id will be also present thus building organization-level features would be possible.

With the MVP, we would be able to migrate the existing contribution analytics, AI impact analytics to use the new data (this is one of the main goal of the MVP). Here, option for organization-level querying is possible (your point 2).

Additionally, counting Projects, Issues, MergeRequests is also possible to some extent (extra effort is needed to efficiently structure data for org/group/project level querying). (point 1).

We would probably see production deployment of Siphon around Q1 if things go well. Some prep work has already started:

• gitlab-org/analytics-section/siphon#26 (closed)
• gitlab-org/analytics-section/siphon#51 (closed) (this would pretty much cover your point 2)

For point 1, we need to do some research to ensure counting of items is efficient on the org, group, project level.

Old proposal

Once the research on &16260 (closed) will be completed:

1. Extended the metrics comparison panel to queries data from CH.
2. The first MVC will include these metrcis: Lead time, New issues, Closed issues and MR Throughput.
3. The panel timespan - 180 days.
4. For each metric displays MtD, Last full calendar month, and Previous full calendar month, with Change % badge.
5. The drill-down will be to the top-group.
6. GitLab Ultimate

Problem Validation

Reference for Customer feedback :

• #413202 (comment 1980742928)
• Instance Level Analytics (with group level data) (#12077 - closed)
• https://docs.google.com/document/d/1TVaQcVj2Fq1-v4urgqz8OepUSucJH4j-FxsbicnGp6Q/edit?tab=t.0
• https://gitlab.com/gitlab-com/account-management/eastern-north-america/southwest-airlines/southwest-airlines-collab/-/issues/266#note_1981541858

Known limitations:

1. Start with SaaS only.
2. Self-managed will be added later as BYO CH.

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.