dev_section.yml 45.9 KB
Newer Older
Eric Brinkman's avatar
Eric Brinkman committed
1
- name: Dev - Section TMAU - Estimated & Recorded Sum of Dev Section SMAU
2 3
  base_path: "/handbook/product/performance-indicators/"
  definition: A sum of all SMAUs across the three stages in the Dev Section (Manage,
Jerome Ng's avatar
Jerome Ng committed
4
    Plan, and Create) for All users and Paid users.
Eric Brinkman's avatar
Eric Brinkman committed
5
  target: Estimated TMAU growth 4% MoM; 2.73M by end of FY21Q4 for TMAU, 603K by end of FY21Q4 for Paid TMAU
6
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
7
  section: dev
8
  public: true
Jerome Ng's avatar
Jerome Ng committed
9
  pi_type: Section TMAU
10
  product_analytics_type: Both
11 12 13
  is_primary: true
  is_key: false
  health:
14
    level: 3
15
    reasons:
David DeSanto's avatar
David DeSanto committed
16
    - TMAU is fully instrumented
David DeSanto's avatar
David DeSanto committed
17 18 19 20 21 22
    - Dev Section Jan 2021 TMAU remained flat MoM and remained down 3.5% from Nov 2020
    - Dev Section exceeded TMAU target of 2.73M by 1.4% by end of FY21 
    - Dev Section Jan 2021 SaaS TMAU bounced back from its Dec 2021 value
    - Dev Section Jan 2021 Self-Managed TMAU continues to grow MoM 
    - Plan Stage Jan 2021 TMAU increased and exceeded its previous highest TMAU value
    - Create Stage Jan 2021 TMAU increased and is trending upwards
23
    - Manage Stage Jan 2021 TMAU decreased MoM
Jerome Ng's avatar
Jerome Ng committed
24
  implementation:
25
    status: Complete
Eric Brinkman's avatar
Eric Brinkman committed
26
    reasons:
David DeSanto's avatar
David DeSanto committed
27 28 29 30 31 32 33 34 35 36 37 38 39
    lessons:
      learned:
      - Dev Section Jan 2021 TMAU remained flat MoM and remained down 3.5% from Nov 2020
      - Dev Section exceeded TMAU target of 2.73M by 1.4% by end of FY21 
      - Dev Section Jan 2021 SaaS TMAU bounced back from its Dec 2021 value
      - Dev Section Jan 2021 Self-Managed TMAU continues to grow MoM 
      - Plan Stage Jan 2021 TMAU increased and exceeded its previous highest TMAU value
      - Create Stage Jan 2021 TMAU increased and is trending upwards
      - Manage Stage Jan 2021 TMAU decreaded MoM 
  monthly_focus:
    goals:
      - Work with Data team to better calculate "Estimated Self-Managed Uplift" to help balance out calculations
      - Analyze Manage Stage TMAU drop and formulate plan to recapture lost TMAU
40
  metric_name: issues, action_monthly_active_users_project_repo, analytics_unique_visits_for_any_target
Eric Brinkman's avatar
Eric Brinkman committed
41
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
42 43
    chart: 9982128
    dashboard: 758607
44 45
    embed: v2
  sisense_data_secondary:
46 47
    chart: 10138877
    dashboard: 758607
Eric Brinkman's avatar
Eric Brinkman committed
48
    embed: v2
Mathieu Peychet's avatar
Mathieu Peychet committed
49 50 51 52
  sisense_data_tertiary:
    chart: 9992267
    dashboard: 758607
    embed: v2
53

Jeremy Watson (ex-GitLab)'s avatar
Jeremy Watson (ex-GitLab) committed
54
- name: Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features
55
  base_path: "/handbook/product/performance-indicators/"
56
  definition: The number of unique users viewing any <a href="https://docs.gitlab.com/ee/user/analytics/">Analytics page</a>.
57
  target: 116,050
58
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
59
  section: dev
Jerome Ng's avatar
Jerome Ng committed
60
  stage: manage
61
  group: analytics
62
  public: true
63
  pi_type: SMAU, GMAU
64
  product_analytics_type: Both
65
  is_primary: true
66 67
  is_key: false
  health:
68
    level: 3
69
    reasons:
70
    - Insights - The Optimize data for January is inaccurate due to [this usage ping issue](https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/290). Many page views for self managed are showing as negative.
Larissa Lane's avatar
Larissa Lane committed
71 72 73
    - Insights - Page views by SaaS users continued a downward trend that started in December. SaaS usage dropped by 10% in December, and again by 13% in January. Views are down for all Optimize pages except Group Insights, which increased by 45%.
    - Improvements - The new DevOps Adoption feature was rolled back in January to pivot from the concept of Segments to only supporting groups in the table (https://gitlab.com/gitlab-org/gitlab/-/issues/296972). There is strong interest in this feature from enterprise customers wanting insights into how GitLab is being used in their organizations. Refactoring work commenced in 13.9 and will continue into 13.10 with the goal of renabling the feature in 13.10 or 13.11. This is expected to result in an increase in DevOps Reports usage. After this work is completed, the feature will be [added at the group level](https://gitlab.com/groups/gitlab-org/-/epics/5252), further increasing availability and adoption.
    - Improvements - Group-level VSA is being refactored to create more space on the page for new features and to simplify navigation. This work is scheduled to be completed in 13.9. Next up we are adding the ability to sort items in each VSA stage by cycle time, which will allow users to identify aging items and take action to reduce overall cycle time in their value streams. Additionally, we are refactoring project-level VSA code to use the group-level code which will improve usability and add functionality when viewing VSA at the project level, as well as allow us to iterate faster because we won't be maintaining two sets of code. All of these improvements combined will improve the usability of VSA and is expected in increase adoption as we announce each improvement. These are the foundational steps we are taking before embarking on our [FY22 vision for VSA](https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10331) which will add many new features and a much richer value stream management experience.
74
    - SaaS data is not available for the paid SMAU chart due to [limitations of the redis counter that is used](https://gitlab.com/gitlab-data/analytics/-/issues/6603#note_484785813).
Jerome Ng's avatar
Jerome Ng committed
75
  implementation:
76
    status: Complete
77
  metric_name: analytics_total_unique_counts_monthly
78
  sisense_data:
Kathleen Tam's avatar
Kathleen Tam committed
79 80
    chart: 10039390
    dashboard: 758607
81
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
82 83 84 85 86
  sisense_data_secondary:
    chart: 10039413
    dashboard: 758607
    embed: v2

87 88 89
- name: Manage:Access - Paid GMAU - MAU Using Paid SAML
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users logging into GitLab using SAML SSO for Groups.
90
  target: 30000. See https://gitlab.com/gitlab-com/Product/-/issues/1832 for further detail.
91
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
92
  section: dev
Jerome Ng's avatar
Jerome Ng committed
93 94
  stage: manage
  group: access
95
  public: true
Jerome Ng's avatar
Jerome Ng committed
96
  pi_type: Paid GMAU
Melissa Ushakov's avatar
Melissa Ushakov committed
97
  product_analytics_type: SaaS
98 99 100
  is_primary: true
  is_key: false
  health:
101
    level: 3
102
    reasons:
103
    - Improvements - In 13.10, Auditor users will have [read-only access to groups with SSO enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/300455). This will allow the support organization to reduce the number of Administrators for GitLab.com and meet their [OKR](https://gitlab.com/gitlab-com/support/managers/okrs/-/issues/29)! 
104 105
    - Improvements - In 13.10, GitLab will begin [enforcing SSO for Git activity](https://gitlab.com/gitlab-org/gitlab/-/issues/22754). This will provide the same level of protection to Git as the Web UI and been requested by several large customers and prospects. 
    - Improvements - In 13.10, we will increase security for GitLab.com by [lowering the timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/298756) for SAML SSO sessions from 7 days to 1 day. 
Melissa Ushakov's avatar
Melissa Ushakov committed
106
    - Insights - We continue to see a healthy upward trend for group SAML logins per month! This is a combination of decreasing the SSO timeout and enteprise customer growth for GitLab.com . My [analysis](https://gitlab.com/gitlab-com/Product/-/issues/2154) shows that decreasing the SSO timeout increased Group SAML usage by 10%.
Jerome Ng's avatar
Jerome Ng committed
107
  implementation:
Kathleen Tam's avatar
Kathleen Tam committed
108
    status: Complete 
109
    reasons:
Melissa Ushakov's avatar
Melissa Ushakov committed
110
  metric_name: user_auth_by_provider (Group_SAML)
Kathleen Tam's avatar
Kathleen Tam committed
111 112 113 114
  sisense_data:
    chart: 10039419
    dashboard: 758607
    embed: v2
115 116 117 118
- name: Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users viewing Audit Events, Compliance Dashboard,
    or Credential Inventory.
119
  target: TBD (expected [April 2021](https://gitlab.com/gitlab-org/manage/compliance/general/-/issues/13))
120
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
121
  section: dev
Jerome Ng's avatar
Jerome Ng committed
122 123
  stage: manage
  group: compliance
124
  public: true
Jerome Ng's avatar
Jerome Ng committed
125
  pi_type: Paid GMAU
126
  product_analytics_type: Both
127 128 129
  is_primary: true
  is_key: false
  health:
130
    level: 2
131
    reasons:
132 133
    - We're still [building out the compliance features](https://gitlab.com/gitlab-org/manage/compliance/general/-/issues/13) that are tracked by this metric. This improvement work will take longer than desired to complete since we do not have enough people to ship any faster. We expect iterations to ship on a continuous basis across each of these features.
    - Setting a target is premature since we're still collecting initial data. We would like to collect a total of 6 months worth of data before setting a target. 
134
    - Result - Increased GMAU by ~25% in December [by making Audit Events available to Developers and making the feature more discoverable](https://gitlab.com/gitlab-org/gitlab/-/issues/257514).
135 136 137
    - Adding a [Change Management view](https://gitlab.com/groups/gitlab-org/-/epics/4802) to the Compliance Dashboard. Planned for `Q2 2021`.
    - Improving PAT & SSH [credential management](https://gitlab.com/groups/gitlab-org/-/epics/3084). Ongoing with expected epic completion in `Q2 2021`.
    - Improving [discoverability](https://gitlab.com/groups/gitlab-org/-/epics/4441) of Compliance features. Planned for `Q2 2021`.
138
        - Adding [additional credentials](https://gitlab.com/groups/gitlab-org/-/epics/4110) to the credential inventory. Planned for `Q3 2021`.
Jerome Ng's avatar
Jerome Ng committed
139
  implementation:
140
    status: Complete 
141
    reasons:
142
  metric_name: i_compliance_audit_events, g_compliance_audit_events, g_compliance_dashboard, i_compliance_credential_inventory, usage_data_g_compliance_dashboard, usage_data_a_compliance_audit_events_api
143
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
144 145
    chart: 9980315
    dashboard: 758607
146
    embed: v2
147 148 149 150
  sisense_data_secondary:
    chart: 10039422
    dashboard: 758607
    embed: v2
151
    
152
- name: Manage:Compliance - Other - Projects Marked with a Compliance Framework
153
  base_path: "/handbook/product/performance-indicators/"
154
  definition: New and cumulative number of projects using compliance frameworks.
155
  target: 1,030 total new projects in Q4 using compliance frameworks (SaaS and self-managed combined)
156
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
157
  section: dev
Jerome Ng's avatar
Jerome Ng committed
158 159
  stage: manage
  group: compliance
160
  public: true
Jerome Ng's avatar
Jerome Ng committed
161
  pi_type: Other
162
  product_analytics_type: Both
163
  is_primary: false
164 165
  is_key: false
  health:
166
    level: 2
167
    reasons:
168 169
    - In Q3 we saw 936 new projects use compliance frameworks. These were a limited selection of 5, hard-coded options. We are targeting a 10% increase in adoption by making these frameworks [customizable](https://gitlab.com/gitlab-org/gitlab/-/issues/255340)
    - Additionally, we're using a conservative target for now while we focus on [several improvements](https://gitlab.com/groups/gitlab-org/-/epics/4082) to the underlying feature that drives this metric.
170 171 172 173
    - We're continuing iteration on [compliance frameworks](https://gitlab.com/groups/gitlab-org/-/epics/4082) to give them more utility and solve several standardization and enforcement problems for customers. 
    - We're adding an improvement to give organizations [better control](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) of who can add or modify these frameworks at the project level in `13.8`.
    - We expect the next iteration, consisting of [frontend](https://gitlab.com/gitlab-org/gitlab/-/issues/292696) and [backend](https://gitlab.com/gitlab-org/gitlab/-/issues/254389) additions to ship in `13.9`. This iteration is particularly important for the reason below.
    - By `13.10` we expect compliance frameworks to fully support one of our most requested solutions, [group compliance pipelines](https://gitlab.com/groups/gitlab-org/-/epics/3156), as a required step to implement it, which we expect to drive even more adoption of compliance frameworks.
Jerome Ng's avatar
Jerome Ng committed
174
  implementation:
175
    status: Complete
176
    reasons:
177
  metric_name: projects_with_compliance_framework
178
  sisense_data:
Matt Gonzales (ex-GitLab)'s avatar
Matt Gonzales (ex-GitLab) committed
179
    chart: 9071082
180 181
    dashboard: 663045
    embed: v2
182

183 184
- name: Manage:Import - GMAU - MAU importing
  base_path: "/handbook/product/performance-indicators/"
185
  definition: A monthly count of the unique number of users performing an import
186
  target: TBD - See Health section below for details
187
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
188
  section: dev
Jerome Ng's avatar
Jerome Ng committed
189 190
  stage: manage
  group: import
191
  public: true
Jerome Ng's avatar
Jerome Ng committed
192
  pi_type: GMAU
193
  product_analytics_type: Both
194
  is_primary: true
195 196 197 198
  is_key: false
  health:
    level: 0
    reasons:
199
    - No target defined yet.
200 201
    - Target will be defined after the [metric instrumentation](https://gitlab.com/gitlab-org/gitlab/-/issues/283175) is implemented and we get a baseline of data.
    - Improvement - Reducing friction for GitLab-to-GitLab migrations - The [MVC for the Group Migration feature](https://gitlab.com/groups/gitlab-org/-/epics/4374) was enabled in production in `13.9`. This feature will enable a much easier migration of an entire GitLab group from one instance to another. 
Jerome Ng's avatar
Jerome Ng committed
202
  implementation:
203
    status: Dashboard
204
    reasons:
205 206 207
    - Instrumentation has been [implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/283175) in `13.9`. 
    - Dashboard [update](https://gitlab.com/gitlab-data/analytics/-/issues/4244) is in progress. 
  metric_name: unique_users_all_imports
208
  
209 210 211 212
- name: Manage:Import - Other - Share of New Projects Imported
  base_path: "/handbook/product/performance-indicators/"
  definition: Share of projects created via import within 90 days of the namespace's
    creation.
213
  target: TBD - See Health section below for details
214
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
215
  section: dev
Jerome Ng's avatar
Jerome Ng committed
216 217
  stage: manage
  group: import
218
  public: true
Jerome Ng's avatar
Jerome Ng committed
219
  pi_type: Other
220
  product_analytics_type: Both
221 222 223 224 225
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
226
    - No target defined yet.
227
    - Target will be defined once the dashboard is [updated](https://gitlab.com/gitlab-data/analytics/-/issues/4244) with the improved metric (for both Saas and self-managed).
228
    - Improvement - Reducing friction for GitLab-to-GitLab migrations - The [MVC for the Group Migration feature](https://gitlab.com/groups/gitlab-org/-/epics/4374) was enabled in production in `13.9`. This feature will enable a much easier migration of an entire GitLab group from one instance to another.
Jerome Ng's avatar
Jerome Ng committed
229
  implementation:
Haris Delalić's avatar
Haris Delalić committed
230
    status: Dashboard
231
    reasons:
232 233
    - Instrumented for [both GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/224758) in `13.6`.
    - Dashboard [update](https://gitlab.com/gitlab-data/analytics/-/issues/4244) is currently in progress.
234
  metric_name: TBD 
Jerome Ng's avatar
Jerome Ng committed
235

236
- name: Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
237 238
  base_path: "/handbook/product/performance-indicators/"
  definition: Number of unique users interacting with an issue (open, close, reopen,
239
    adjust label, adjust milestone, adjust weight, comment) 
240
  target: 484,510 for All, 148,176 for Paid 
241
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
242
  section: dev
Jerome Ng's avatar
Jerome Ng committed
243
  stage: plan
244
  group: project_management
245
  public: true
246
  pi_type: SMAU, Paid GMAU
247
  product_analytics_type: Both
248 249
  is_primary: true
  is_key: false
250 251
  lessons:
    learned:
252
      - Only 28% of instances are currently on a version with the new Plan xMAU metric counters. 
253
      - The ratio of [net new Paid GMAU to churned Paid GMAU](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)?widget=10634701&udv=0) improved from 3.75 in December to 3.99 in January, which indicates net growth and retention for Plan Paid users is healthy.
254
      - MAU interacting with Iterations increased by 52% from December to January. The next step to drive feature adoption is more holistically integrating Iterations into Boards. 
255 256 257
  monthly_focus:
    goals: 
      - Driver - Determine technical feasibility for allowing an [issue to be assigned to multiple groups](https://gitlab.com/gitlab-org/gitlab/-/issues/296668), which would likely solve [Plan adoption blockers (priorities 1 and 3)](https://gitlab.com/gitlab-org/gitlab/-/issues/218333).
258
      - Continue making progress on the [13.9 release plan](https://gitlab.com/gitlab-org/plan/-/issues/236) and roll into [13.10 goals](https://gitlab.com/gitlab-org/plan/-/issues/264).
259
      - Open > 3 documentation MRs in support of [better articulating Plan use cases](https://gitlab.com/gitlab-org/gitlab/-/issues/213810).
260
      - Complete the validation track for [saved filters/searches](https://gitlab.com/gitlab-org/gitlab/-/issues/14512), [customizable issue types](https://gitlab.com/gitlab-org/gitlab/-/issues/118666), and [interacting with a full issue from within a Board](https://gitlab.com/gitlab-org/gitlab/-/issues/13319)
261
      - Merge architectural blueprint for [consolidating Groups and Projects](https://gitlab.com/gitlab-org/architecture/tasks/-/issues/7) on the backend.
262 263 264
  health:
    level: 3
    reasons:
265
    - Achieved - FY21 Q4 targets of 437,490 for All, 101,835 for Paid. New targets set for FY22 Q1. 
266 267
    - Driver - For Paid, continue high level progress on [achieving feature parity between an Instance, Group, and Project](https://gitlab.com/groups/gitlab-org/-/epics/2885), [making issues more extensible](https://gitlab.com/groups/gitlab-org/-/epics/3354), and continuing to make incremental progress on [iterations](https://gitlab.com/groups/gitlab-org/-/epics/2422).
    - Improvement - For All, work towards addressing the [problem areas](https://gitlab.com/groups/gitlab-org/-/epics/4104) that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
268 269
    - Improvement - For All, continue making progress on [quality of life improvements](https://gitlab.com/groups/gitlab-org/-/epics/3216) for existing features. 
    - Improvement - For Paid, creating multi-level Epics spanning a Group hierarchy is clunky and non-intuitive. Improving this, along with making the Roadmap more interactive and intuitive would be a huge catalyst to drive Product Planning adoption. 
270
    - Improvement - For Paid, continue making incremental, iterative progress towards [Plan's core JTBD](/direction/plan/#problem-and-jobs-to-be-done) as outlined in the [Issue Tracking direction](/direction/plan/issue_tracking/). 
271 272
  implementation:
    status: Complete
273
    reasons:
274 275
    - xMAU data models are now updated with the new counter that aggregates unique users across many different issue actions.
    - New [Plan xMAU](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)) dashboard is live.
276
  metric_name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly 
277
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
278 279
    chart: 9967544
    dashboard: 758607
Jerome Ng's avatar
Jerome Ng committed
280 281
    embed: v2
  sisense_data_secondary:
Jerome Ng's avatar
Jerome Ng committed
282 283
    chart: 9980295
    dashboard: 758607
284
    embed: v2
285
  urls:
286
  - https://about.gitlab.com/direction/plan/issue_tracking/
287
  - https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)
Jerome Ng's avatar
Jerome Ng committed
288

Jerome Ng's avatar
Jerome Ng committed
289
- name: Plan:Product Planning - Paid GMAU - MAU creating Epics each Month
290 291
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users creating Epics each month.
292
  target: 3,900 Combined Unique Users for Self Managed and SaaS
293
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
294
  section: dev
Jerome Ng's avatar
Jerome Ng committed
295
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
296
  group: product_planning
297
  public: true
Jerome Ng's avatar
Jerome Ng committed
298
  pi_type: Paid GMAU
299
  product_analytics_type: Both
300 301 302 303 304
  is_primary: true
  is_key: false
  health:
    level: 3
    reasons:
Christen Dybenko's avatar
Christen Dybenko committed
305 306
    - Achieved - Target of 3,900 (3,943 actual GMAU for epic creation) 
    - Insight - [Epic creation](https://app.periscopedata.com/app/gitlab/602355/Portfolio-Management-G-SMAU-and-Feature-Success-Tracking?widget=9245316&udv=962108) for SaaS continues to grow. Self-Managed was stable, but that could be due to the [Janauary tracking bug](https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/290).
307
    - Improvement - [Epic Boards](https://gitlab.com/groups/gitlab-org/-/epics/2864) MVC 1 is planned to ship in 13.10 and should continue the trend into the end of FY21 by enabling Portfolio-level planning.  
308
    - Improvement - [Epic Swimlanes](https://gitlab.com/groups/gitlab-org/-/epics/3352) shipped in 13.6
Christen Dybenko's avatar
Christen Dybenko committed
309
    - Closed - original [PI dashboard issue for epic creation](https://gitlab.com/gitlab-data/analytics/-/issues/5280#note_505497453) 
Jerome Ng's avatar
Jerome Ng committed
310 311
  implementation:
    status: Complete
312
    reasons:
Christen Dybenko's avatar
Christen Dybenko committed
313 314 315
    - We consider our Epic Creation GMAU as "Complete". However, our next target will be set after a second instrumentation.
    - Instrument "all epic interactions" during 13.9 - 13.10.  [Instrumentation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/292253) and [dasboard issue](https://gitlab.com/gitlab-data/analytics/-/issues/7885).
    - A new target will be selected when we have a baseline.
316
  metric_name: epics 
317
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
318 319
    chart: 9980303
    dashboard: 758607
320 321
    embed: v2
  urls:
Christen Dybenko's avatar
Christen Dybenko committed
322
    
Mark Wood's avatar
Mark Wood committed
323
- name: Plan:Certify - Paid GMAU - MAU Interacting with Requirements
324 325 326 327 328
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users interacting (create, view, reference) each
    month
  target: TBD
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
329
  section: dev
Jerome Ng's avatar
Jerome Ng committed
330
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
331
  group: product_planning
332
  public: true
Jerome Ng's avatar
Jerome Ng committed
333
  pi_type: Paid GMAU
334
  product_analytics_type: Both
Jerome Ng's avatar
Jerome Ng committed
335
  is_primary: false
336 337
  is_key: false
  health:
Jerome Ng's avatar
Jerome Ng committed
338
    level: 0
339
    reasons:
Mark Wood's avatar
Mark Wood committed
340 341 342
    - Note - [Requirement interaction tracking](https://gitlab.com/gitlab-org/product-intelligence/-/issues/370) issue is complete. 
      Data is in the process of being aggregated. Early data shows healthy adoption of test report linking.
      [Tracking issue](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
343
    - Improvement - UX improvements for [Requirements Management](https://gitlab.com/groups/gitlab-org/-/epics/3708)
Mark Wood's avatar
Mark Wood committed
344
      based on completed research study, Requirement [Export MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/290813).
Jerome Ng's avatar
Jerome Ng committed
345 346
  implementation:
    status: Dashboard
347
    reasons:
Justin Farris's avatar
Justin Farris committed
348
    - Dashboarding - Requirement Creation is instrumented is complete. Tracking issue for gmau dashboard [here](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
349
  metric_name: TBD
Mark Wood's avatar
Mark Wood committed
350
- name: Plan:Certify - Other - Number of Requirements Created Monthly
351 352
  base_path: "/handbook/product/performance-indicators/"
  definition: A monthly count of the number of requirements created.
Mark Wood's avatar
Mark Wood committed
353
  target: 1000 by end of March 2021
354
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
355
  section: dev
Jerome Ng's avatar
Jerome Ng committed
356
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
357
  group: product_planning
358
  public: true
Jerome Ng's avatar
Jerome Ng committed
359
  pi_type: Other
360
  product_analytics_type: Both
361 362 363 364 365
  is_primary: false
  is_key: false
  health:
    level: 3
    reasons:
Mark Wood's avatar
Mark Wood committed
366 367
    - Improvement - [Requirement Export MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/290813) feature to release in 13.8
    - Insight - Requirement creation has decreased. Investigation shows a substantial set of users are Universities, which explains the downturn for the holidays. 
Mark Wood's avatar
Mark Wood committed
368
    - Insight - Self-managed instances creating requirements continues to grow.   
Jerome Ng's avatar
Jerome Ng committed
369 370
  implementation:
    status: Dashboard
371
    reasons:
372
    - Dashboarding -  accurate instrumentation on Self Managed, still needs verification
373
      for SaaS - [issue](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
Jerome Ng's avatar
Jerome Ng committed
374
  metric_name: requirements_created
375
  sisense_data:
Mark Wood's avatar
Mark Wood committed
376
    chart: 9666808
377
    dashboard: 634539
378 379 380
    embed: v2
  urls:
  - https://gitlab.com/gitlab-data/analytics/-/issues/5270
381

James Ramsay's avatar
James Ramsay committed
382
- name: Create - SMAU - Unique users writing to a Git repository
383
  base_path: "/handbook/product/performance-indicators/"
384 385
  definition: A rolling count of the number of unique users who performed any Git write operation within the last 28 days.
  target: |
James Ramsay's avatar
James Ramsay committed
386
    Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
387
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
388
  section: dev
Jerome Ng's avatar
Jerome Ng committed
389
  stage: Create
390
  public: true
Jerome Ng's avatar
Jerome Ng committed
391
  pi_type: SMAU
392
  product_analytics_type: Both
393 394 395
  is_primary: true
  is_key: false
  health:
396
    level: 3
397
    reasons:
James Ramsay's avatar
James Ramsay committed
398 399 400 401 402
    - Observation - December 2020 SMAU is less than November 2020 SMAU. A
      similar drop can be observed in GMAU as well, suggesting a common cause.
      The cause is likely seasonality consistent with the end of year. For this
      reason January SMAU and GMAU is expected to return to November 2020
      levels and continue growing.
James Ramsay's avatar
James Ramsay committed
403 404 405 406 407 408
    - Insight - on the median GitLab instance, 33% of users perform a Git write
      operation each month
      ([source](https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=10435358&udv=1100338)).
    - Insight - on the median Paid GitLab instance (self-managed), 50% of users
      perform a Git write operation each month
      ([source](https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=10435361&udv=1100338)).
Jerome Ng's avatar
Jerome Ng committed
409 410
  implementation:
    status: Complete
411
    reasons:
James Ramsay's avatar
James Ramsay committed
412 413 414 415
    - Improvement - AMAU is currently over counted. A deduplicated counter was implemented
      [gitlab#234027](https://gitlab.com/gitlab-org/gitlab/-/issues/234027) in GitLab 13.6.
      The charts do not yet reflect the new counter
      [gitlab#291943](https://gitlab.com/gitlab-org/gitlab/-/issues/291943).
416 417
  metric_name: action_monthly_active_users_project_repo
  sisense_data:
Kathleen Tam's avatar
Kathleen Tam committed
418 419
    chart: 9967642
    dashboard: 758607
420
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
421 422 423 424
  sisense_data_secondary:
    chart: 9980442
    dashboard: 758607
    embed: v2
425
  urls:
Daniel Gruesso's avatar
Daniel Gruesso committed
426
  - https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard
427

James Ramsay's avatar
James Ramsay committed
428
- name: Create:Source Code - GMAU - Unique users writing to a project Git repository
429
  base_path: "/handbook/product/performance-indicators/"
430 431 432
  definition: A rolling count of the number of unique users who performed any Git write
    operation within the last 28 days
  target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
433
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
434
  section: dev
Jerome Ng's avatar
Jerome Ng committed
435 436
  stage: create
  group: source_code
437
  public: true
Jerome Ng's avatar
Jerome Ng committed
438
  pi_type: GMAU
439
  product_analytics_type: Both
440 441 442
  is_primary: true
  is_key: false
  health:
Daniel Gruesso's avatar
Daniel Gruesso committed
443
    level: 3
444
    reasons:
445
    - Starting in [GitLab 13.6](https://gitlab.com/gitlab-org/gitlab/-/issues/234027)
446
      AMAU for this metric takes into consideration all repository types.
447 448 449 450
    - Improvement - work towards providing users more insights while navigating their
      code with [Code Intelligence](https://gitlab.com/groups/gitlab-org/-/epics/1576)
    - Improvement - work towards providing more control over [managing repository
      size](https://gitlab.com/groups/gitlab-org/-/epics/3793)
Jerome Ng's avatar
Jerome Ng committed
451
  implementation:
452
    status: Complete
453
    reasons:
454
    - Enabled on GitLab.com
455 456
    - Released for self-managed in GitLab 13.3
  metric_name: action_monthly_active_users_project_repo
Daniel Gruesso's avatar
Daniel Gruesso committed
457
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
458 459 460 461 462 463
    chart: 9967642
    dashboard: 758607
    embed: v2
  sisense_data_secondary:
    chart: 9980442
    dashboard: 758607
Daniel Gruesso's avatar
Daniel Gruesso committed
464
    embed: v2
465
  urls:
Daniel Gruesso's avatar
Daniel Gruesso committed
466
  - https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=9520571&udv=0
James Ramsay's avatar
James Ramsay committed
467
  - https://gitlab.com/gitlab-org/gitlab/-/issues/234027
468

469
- name: Create:Source Code Secondary - GMAU - Unique users approving merge requests
470
  base_path: "/handbook/product/performance-indicators/"
471
  definition: A rolling count of the number of unique users who approved any merge request within the last 28 days
472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: source_code
  public: true
  pi_type: GMAU
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Not currently instrumented
    - Currently planning instrumentation and evaluating if this is a more relevant metric for the Source Code group
  urls:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/293014

489
- name: Create:Source Code Tertiary - GMAU - Unique users pushes rejected by push rules
490
  base_path: "/handbook/product/performance-indicators/"
491
  definition: A rolling count of the number of unique users who had one or more push rejected by any push rule within the last 28 days
492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: source_code
  public: true
  pi_type: GMAU
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Not currently instrumented
    - Currently planning instrumentation and evaluating if this is a more relevant metric for the Source Code group
  urls:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/293016

James Ramsay's avatar
James Ramsay committed
509
- name: Create:Code Review - GMAU - Unique users using Merge Requests
510
  base_path: "/handbook/product/performance-indicators/"
511 512
  definition: This is defined as users who perform one or more of the following actions in
   a merge request - create, merge, or close a Merge Request.
Kai Armstrong's avatar
Kai Armstrong committed
513
  target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU. 
514
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
515
  section: dev
Jerome Ng's avatar
Jerome Ng committed
516
  stage: create
517
  group: code_review
518
  public: true
James Ramsay's avatar
James Ramsay committed
519
  pi_type: GMAU
520
  product_analytics_type: Both
521
  is_primary: true
522 523
  is_key: false
  health:
524
    level: 3
525
    reasons:
Kai Armstrong's avatar
Kai Armstrong committed
526
    - Target - This aligns with our current understanding that Code Review is about 50% of the MAU of the stage metric and continuing to maintain that difference.
527
    - Insight - Paid users of Code Review make up about one-third of total Code Review users - this provides an opportunity to convert to paid and increase users.
528
    - Improvement - Support for [custom commit messages when applying a suggested change](https://gitlab.com/gitlab-org/gitlab/-/issues/25381) will allow more teams with strict commit message guidelines to adopt this feature and maintain conformity with the project.
529 530
    - Improvement - [Mark changes as viewed](https://gitlab.com/gitlab-org/gitlab/-/issues/17531/) in merge requests is a foundational step towards tracking and communicating progress on merge requests to other participants.
    - Improvement - SCM <-> Code Review <-> Verify correlation charts now defined in [new charts](#createcode-review---other---code-reviewscm-ratio-by-umau-and-subscription-tier)
Jerome Ng's avatar
Jerome Ng committed
531
  implementation:
532
    status: Complete
533
    reasons:
534 535
    - Enabled on GitLab.com 2020-07-04
    - Enabled for self-managed on GitLab 13.2
536
    - Working on more comprehensive MAU tracking in [&4948](https://gitlab.com/groups/gitlab-org/-/epics/4948)
537
  metric_name: merge_requests_users 
Daniel Gruesso's avatar
Daniel Gruesso committed
538
  sisense_data:
539 540
    chart: 10426645
    dashboard: 786738
Daniel Gruesso's avatar
Daniel Gruesso committed
541
    embed: v2
542
  sisense_data_secondary:
543 544
    chart: 10624715
    dashboard: 786738
545
    embed: v2
546
  urls:
547
  - https://app.periscopedata.com/app/gitlab/526749/Create---Code-Review-category-dashboard
James Ramsay's avatar
James Ramsay committed
548

549
- name: Create:Code Review - Other - Code Review/SCM Ratio by UMAU and Subscription Tier
Kai Armstrong's avatar
Kai Armstrong committed
550 551
  base_page: "/handbook/product/performance-indicators/"
  definition: 
552
  target: TBD - Primarily this metric is used for monitoring the relationship between SCM GMAU and Code Review GMAU to make sure it doesn't not fall out of alignment.
Kai Armstrong's avatar
Kai Armstrong committed
553 554 555 556 557 558 559 560 561 562 563 564 565
  org: Dev Section
  section: dev
  stage: create
  group: code_review
  public: true
  pi_type: Other
  product_analytics_type: Both
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Target - this metric is primarily monitored to remain at a consistent level.
566 567
    - Insight - Paid Instances with 10+ users more closely track to 75% of SCM Users using Code Review
    - Insight - Free Instances and those with less than 10 users drive down the SCM<->Code Review Ratio as most have less than 50% adoption
Kai Armstrong's avatar
Kai Armstrong committed
568 569 570 571
  implementation:
    status: Complete
  metric_name: merge_requests_users
  sisense_data:
572 573 574 575 576 577
    chart: 10728688
    dashboard: 801102
    embed: v2
  sisense_data_secondary:
    chart: 10688270
    dashboard: 801102
Kai Armstrong's avatar
Kai Armstrong committed
578
    embed: v2
579 580
  urls:
  - https://app.periscopedata.com/app/gitlab/801102/SCM-%3C%3E-Code-Review-%3C%3E-Verify
581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605
  - https://app.periscopedata.com/app/gitlab/801102/SCM-%3C%3E-Code-Review-%3C%3E-Verify?widget=10728688&udv=1218043

- name: Create:Code Review - Editor Extension (VS Code) - Category MAU
  base_path: "/handbook/product/performance-indicators/"
  definition: This is defined as users who use [GitLab Workflow](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow) for VS Code and make at least one API request to a GitLab instance.
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: code_review
  public: true
  pi_type: Other
  product_analytics_type: Both
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Instrumentation in 13.9
    - Improvement - [GitLab Workflow 3.11.0](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CHANGELOG.md#3110-2021-01-25) released with auto completion for GitLab CI Variables
    - Improvement - [GitLab Workflow 3.10.0](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CHANGELOG.md#3100-2021-01-19) released with viewing code review comments on merge request diffs
  implementation:
    status: Instrumentation
    reasons:
    - MAU Collection [Issue](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/issues/258)
Kai Armstrong's avatar
Kai Armstrong committed
606

607
- name: Create:Gitaly - Other - Availability
608
  base_path: "/handbook/product/performance-indicators/"
609 610 611 612 613 614 615
  definition: Percentage of time during which Gitaly is fully operational and providing
    service to users within SLO parameters. Availability is measured according by
    Apdex (latency) and error rate. Gitaly is considered available when at least 50%
    of users are experiencing satisfactory latency, _and_ 50% of requests are completing
    successfully. If either of these conditions is not met, the service is experiencing
    an outage.
  target: 99.95%, consistent with GitLab.com availability service level objectives
616
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
617
  section: dev
618 619
  stage: create
  group: gitaly
620
  public: true
Jeremy Watson (ex-GitLab)'s avatar
Jeremy Watson (ex-GitLab) committed
621
  pi_type: Other
622
  product_analytics_type: Both
623 624 625 626 627 628 629 630 631 632
  is_primary: true
  is_key: false
  health:
    level: 0
    reasons:
    - Instrumentation for this metric started in 13.4 and is still ongoing.
  implementation:
    status: Instrumentation
    reasons:
    - Rolling 7 day metrics available in [Grafana](https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview)
Mark Wood's avatar
Mark Wood committed
633
      for GitLab.com. Current availability per Adpex is 99.97%.
634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650
    - Improvement - self-managed availability metrics are [planned](https://gitlab.com/gitlab-org/gitlab/-/issues/216683)
  metric_name: gitaly_apdex
  urls:
  - https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
  - https://gitlab.com/gitlab-org/gitlab/-/issues/216683

- name: Create:Gitaly - Other - Unique instances with Gitaly Cluster enabled
  base_path: "/handbook/product/performance-indicators/"
  definition: Number of recorded GitLab instances with Gitaly Cluster enabled
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: gitaly
  public: true
  pi_type: Other
  product_analytics_type: Self-Managed
Christen Dybenko's avatar
Christen Dybenko committed
651
  is_primary: false
652 653
  is_key: false
  health:
654
    level: 2
655
    reasons:
656 657
    - No target defined as the feature is still being rolled out.
    - Improvement - introduce [Variable Replication Factor and auto-reblancing within a cluster](https://gitlab.com/groups/gitlab-org/-/epics/3372) to enable more enterprise-level adoption.
Jerome Ng's avatar
Jerome Ng committed
658
  implementation:
659
    status: Complete
660
    reasons:
Mark Wood's avatar
Mark Wood committed
661
    - Insight - Seeing impressive growth in Gitaly Cluster adoption.
662 663
    - Instrumentation for reporting on instances with Gitaly Cluster enabled was introduced in 13.0 with the GA of Gitaly Cluster.
  metric_name: gitaly_clusters
Jerome Ng's avatar
Jerome Ng committed
664
  sisense_data:
665 666
    chart: 9504807
    dashboard: 728407
Jerome Ng's avatar
Jerome Ng committed
667
    embed: v2
668
  urls:
669
  - https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
James Ramsay's avatar
James Ramsay committed
670 671

- name: Create:Editor - GMAU - Unique users committing via any web editor
672 673
  base_path: "/handbook/product/performance-indicators/"
  definition: A rolling count of the number of unique users who performed a Git write
674 675
    operation to any respository using the Web IDE, Web Editor or Snippet within a
    stage in the last 28 days
Eric Schurter's avatar
Eric Schurter committed
676
  target: 4% growth month over month for All 
677
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
678
  section: dev
Jerome Ng's avatar
Jerome Ng committed
679 680
  stage: create
  group: editor
681
  public: true
Jerome Ng's avatar
Jerome Ng committed
682
  pi_type: GMAU
683
  product_analytics_type: Both
684
  is_primary: true
685 686
  is_key: false
  health:
687
    level: 2
688
    reasons:
689
    - New target GMAU to be set once we get cumulative, de-duped data from newly aligned categories.
690 691
    - Insight - Rebound from the holiday season to nearly November levels. We'll be monitoring to ensure consistent organic growth in subsequent months. 
    - Improvement - No targeted efforts in the next few releases to specifically grow GMAU, but currently working to [migrate the Web IDE to use Editor Lite](https://gitlab.com/gitlab-org/gitlab/-/issues/297449) which will bring consistency across editors and a [moderate performance improvement](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51212#note_483338464). 
Jerome Ng's avatar
Jerome Ng committed
692
  implementation:
Jerome Ng's avatar
Jerome Ng committed
693
    status: Complete
694
    reasons:
695
    - Instrumentation of [Static Site Editor and Wiki to roll up into GMAU](https://gitlab.com/gitlab-org/gitlab/-/issues/283947) completed in 13.7, waiting for data to be reflected in the charts.
696
  metric_name: action_monthly_active_users_ide_edit 
697
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
698 699
    chart: 9967737
    dashboard: 758607
700
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
701 702 703
  sisense_data_secondary:
    chart: 9980340
    dashboard: 758607
Jerome Ng's avatar
Jerome Ng committed
704
    embed: v2
705 706
  urls:
  - https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU
James Ramsay's avatar
James Ramsay committed
707 708

- name: Create:Editor - AMAU - Unique users committing by web editor
709 710 711 712 713 714