dev_section.yml 47.2 KB
Newer Older
Eric Brinkman's avatar
Eric Brinkman committed
1
- name: Dev - Section TMAU - Estimated & Recorded Sum of Dev Section SMAU
2 3
  base_path: "/handbook/product/performance-indicators/"
  definition: A sum of all SMAUs across the three stages in the Dev Section (Manage,
Jerome Ng's avatar
Jerome Ng committed
4
    Plan, and Create) for All users and Paid users.
Eric Brinkman's avatar
Eric Brinkman committed
5
  target: Estimated TMAU growth 4% MoM; 2.73M by end of FY21Q4 for TMAU, 603K by end of FY21Q4 for Paid TMAU
6
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
7
  section: dev
8
  public: true
Jerome Ng's avatar
Jerome Ng committed
9
  pi_type: Section TMAU
10
  product_analytics_type: Both
11 12 13
  is_primary: true
  is_key: false
  health:
14
    level: 3
15
    reasons:
Eric Brinkman's avatar
Eric Brinkman committed
16 17 18
    - We saw a 3.5% drop in Dev Section TMAU between Nov 2020 and Dec 2020. UMAU dropped 5.2% as a reference point.
    - Despite the drop, the Dev section has already achieved our end of Q4 target of 2.75M TMAU by 300K users.
    - TMAU drops were the largest for Create. Plan declined slightly by 3K users and Manage increased by 8K users, most likely because of upgrades as the GMAU was only turned on by default in 13.5.
19
    - Self-managed TMAU increased, but SaaS significantly decreased, likely to free users declining due to seasonality - as a next step, we'll work to add a 4th graph in this section with TMAU broken out by tiers, including free and paid.
Eric Brinkman's avatar
Eric Brinkman committed
20 21
    - Insight - instances with large amounts of users are unlikely to upgrade immediately, typically taking a few months to do so. So, our larger instances were likely counted as estimated earlier and as the larger instances upgrade, what we'll likely see is that estimated uplift was under reported early. It's difficult to tell if it will be over reported as our largest self-managed instances are unlikely to ever report back. It will be interesting to see how the uplifts change when we base the uplift on users instead of subscriptions.
    - Improvement - For All Usage, address the [problems](https://gitlab.com/groups/gitlab-org/-/epics/4104)
22
      that emerged out of the Category Maturity Scorecard exercise for Issue Tracking
Eric Brinkman's avatar
Eric Brinkman committed
23
      and continue making progress on [usability improvements](https://gitlab.com/groups/gitlab-org/-/epics/3216)
24
      for existing Plan features.
Eric Brinkman's avatar
Eric Brinkman committed
25 26 27 28 29 30
    - Improvement - For All Usage, Merge Request [usability improvements](https://gitlab.com/groups/gitlab-org/-/epics/3607)
    - Improvement - For All Usage, [MR reviewers](https://gitlab.com/gitlab-org/gitlab/-/issues/237921) will make us more competitive with GitHub and should help with the lovability of assigning merge requests
    - Improvement - For All Usage, [Iterations in GitLab](https://gitlab.com/groups/gitlab-org/-/epics/2422) will make plan more usable for many Agile development teams
    - Improvement - For Paid Usage, [Epic Boards](https://gitlab.com/groups/gitlab-org/-/epics/4117) will allow for much better portfolio management via boards
    - Improvement - For Paid Usage, [SSO Group Sync](https://gitlab.com/gitlab-org/gitlab/-/issues/118)
    - Improvement - For All Usage, [Workspaces](https://gitlab.com/groups/gitlab-org/-/epics/4257)
Eric Brinkman's avatar
Eric Brinkman committed
31 32
    - Improvement - For All Usage, [Groups/Project Simplification](https://gitlab.com/groups/gitlab-org/-/epics/2885)
    - Improvement - For All Usage, [Settings and Navigation improvements](https://gitlab.com/groups/gitlab-org/-/epics/4831)
Jerome Ng's avatar
Jerome Ng committed
33
  implementation:
34
    status: Complete
Eric Brinkman's avatar
Eric Brinkman committed
35
    reasons:
36
  metric_name: issues, action_monthly_active_users_project_repo, analytics_unique_visits_for_any_target
Eric Brinkman's avatar
Eric Brinkman committed
37
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
38 39
    chart: 9982128
    dashboard: 758607
40 41
    embed: v2
  sisense_data_secondary:
42 43
    chart: 10138877
    dashboard: 758607
Eric Brinkman's avatar
Eric Brinkman committed
44
    embed: v2
Mathieu Peychet's avatar
Mathieu Peychet committed
45 46 47 48
  sisense_data_tertiary:
    chart: 9992267
    dashboard: 758607
    embed: v2
49

Jeremy Watson (ex-GitLab)'s avatar
Jeremy Watson (ex-GitLab) committed
50
- name: Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features
51
  base_path: "/handbook/product/performance-indicators/"
52
  definition: The number of unique users viewing any <a href="https://docs.gitlab.com/ee/user/analytics/">Analytics page</a>.
53
  target: 116,050
54
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
55
  section: dev
Jerome Ng's avatar
Jerome Ng committed
56
  stage: manage
57
  group: analytics
58
  public: true
59
  pi_type: SMAU, GMAU
60
  product_analytics_type: Both
61
  is_primary: true
62 63
  is_key: false
  health:
64
    level: 3
65
    reasons:
66
    - Insights - The Optimize data for January is inaccurate due to [this usage ping issue](https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/290). Many page views for self managed are showing as negative.
Larissa Lane's avatar
Larissa Lane committed
67 68 69
    - Insights - Page views by SaaS users continued a downward trend that started in December. SaaS usage dropped by 10% in December, and again by 13% in January. Views are down for all Optimize pages except Group Insights, which increased by 45%.
    - Improvements - The new DevOps Adoption feature was rolled back in January to pivot from the concept of Segments to only supporting groups in the table (https://gitlab.com/gitlab-org/gitlab/-/issues/296972). There is strong interest in this feature from enterprise customers wanting insights into how GitLab is being used in their organizations. Refactoring work commenced in 13.9 and will continue into 13.10 with the goal of renabling the feature in 13.10 or 13.11. This is expected to result in an increase in DevOps Reports usage. After this work is completed, the feature will be [added at the group level](https://gitlab.com/groups/gitlab-org/-/epics/5252), further increasing availability and adoption.
    - Improvements - Group-level VSA is being refactored to create more space on the page for new features and to simplify navigation. This work is scheduled to be completed in 13.9. Next up we are adding the ability to sort items in each VSA stage by cycle time, which will allow users to identify aging items and take action to reduce overall cycle time in their value streams. Additionally, we are refactoring project-level VSA code to use the group-level code which will improve usability and add functionality when viewing VSA at the project level, as well as allow us to iterate faster because we won't be maintaining two sets of code. All of these improvements combined will improve the usability of VSA and is expected in increase adoption as we announce each improvement. These are the foundational steps we are taking before embarking on our [FY22 vision for VSA](https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10331) which will add many new features and a much richer value stream management experience.
70
    - SaaS data is not available for the paid SMAU chart due to [limitations of the redis counter that is used](https://gitlab.com/gitlab-data/analytics/-/issues/6603#note_484785813).
Jerome Ng's avatar
Jerome Ng committed
71
  implementation:
72
    status: Complete
73
  metric_name: analytics_total_unique_counts_monthly
74
  sisense_data:
Kathleen Tam's avatar
Kathleen Tam committed
75 76
    chart: 10039390
    dashboard: 758607
77
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
78 79 80 81 82
  sisense_data_secondary:
    chart: 10039413
    dashboard: 758607
    embed: v2

83 84 85
- name: Manage:Access - Paid GMAU - MAU Using Paid SAML
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users logging into GitLab using SAML SSO for Groups.
86
  target: 30000. See https://gitlab.com/gitlab-com/Product/-/issues/1832 for further detail.
87
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
88
  section: dev
Jerome Ng's avatar
Jerome Ng committed
89 90
  stage: manage
  group: access
91
  public: true
Jerome Ng's avatar
Jerome Ng committed
92
  pi_type: Paid GMAU
Melissa Ushakov's avatar
Melissa Ushakov committed
93
  product_analytics_type: SaaS
94 95 96
  is_primary: true
  is_key: false
  health:
97
    level: 3
98
    reasons:
99
    - Improvements - In 13.10, Auditor users will have [read-only access to groups with SSO enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/300455). This will allow the support organization to reduce the number of Administrators for GitLab.com and meet their [OKR](https://gitlab.com/gitlab-com/support/managers/okrs/-/issues/29)! 
100 101 102
    - Improvements - In 13.10, GitLab will begin [enforcing SSO for Git activity](https://gitlab.com/gitlab-org/gitlab/-/issues/22754). This will provide the same level of protection to Git as the Web UI and been requested by several large customers and prospects. 
    - Improvements - In 13.10, we will increase security for GitLab.com by [lowering the timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/298756) for SAML SSO sessions from 7 days to 1 day. 
    - Insights - We continue to see a healthy upward trend for group SAML logins per month! This is a combination of decreasing the SSO timeout and enteprise customer growth for GitLab.com . 
Jerome Ng's avatar
Jerome Ng committed
103
  implementation:
Kathleen Tam's avatar
Kathleen Tam committed
104
    status: Complete 
105
    reasons:
Melissa Ushakov's avatar
Melissa Ushakov committed
106
  metric_name: user_auth_by_provider (Group_SAML)
Kathleen Tam's avatar
Kathleen Tam committed
107 108 109 110
  sisense_data:
    chart: 10039419
    dashboard: 758607
    embed: v2
111 112 113 114
- name: Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users viewing Audit Events, Compliance Dashboard,
    or Credential Inventory.
115
  target: TBD (expected [April 2021](https://gitlab.com/gitlab-org/manage/compliance/general/-/issues/13))
116
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
117
  section: dev
Jerome Ng's avatar
Jerome Ng committed
118 119
  stage: manage
  group: compliance
120
  public: true
Jerome Ng's avatar
Jerome Ng committed
121
  pi_type: Paid GMAU
122
  product_analytics_type: Both
123 124 125
  is_primary: true
  is_key: false
  health:
126
    level: 2
127
    reasons:
128 129
    - We're still [building out the compliance features](https://gitlab.com/gitlab-org/manage/compliance/general/-/issues/13) that are tracked by this metric. This improvement work will take longer than desired to complete since we do not have enough people to ship any faster. We expect iterations to ship on a continuous basis across each of these features.
    - Setting a target is premature since we're still collecting initial data. We would like to collect a total of 6 months worth of data before setting a target. 
130
    - Result - Increased GMAU by ~25% in December [by making Audit Events available to Developers and making the feature more discoverable](https://gitlab.com/gitlab-org/gitlab/-/issues/257514).
131 132 133
    - Adding a [Change Management view](https://gitlab.com/groups/gitlab-org/-/epics/4802) to the Compliance Dashboard. Planned for `Q2 2021`.
    - Improving PAT & SSH [credential management](https://gitlab.com/groups/gitlab-org/-/epics/3084). Ongoing with expected epic completion in `Q2 2021`.
    - Improving [discoverability](https://gitlab.com/groups/gitlab-org/-/epics/4441) of Compliance features. Planned for `Q2 2021`.
134
        - Adding [additional credentials](https://gitlab.com/groups/gitlab-org/-/epics/4110) to the credential inventory. Planned for `Q3 2021`.
Jerome Ng's avatar
Jerome Ng committed
135
  implementation:
136
    status: Complete 
137
    reasons:
138
  metric_name: i_compliance_audit_events, g_compliance_audit_events, g_compliance_dashboard, i_compliance_credential_inventory, usage_data_g_compliance_dashboard, usage_data_a_compliance_audit_events_api
139
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
140 141
    chart: 9980315
    dashboard: 758607
142
    embed: v2
143 144 145 146
  sisense_data_secondary:
    chart: 10039422
    dashboard: 758607
    embed: v2
147
    
148
- name: Manage:Compliance - Other - Projects Marked with a Compliance Framework
149
  base_path: "/handbook/product/performance-indicators/"
150
  definition: New and cumulative number of projects using compliance frameworks.
151
  target: 1,030 total new projects in Q4 using compliance frameworks (SaaS and self-managed combined)
152
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
153
  section: dev
Jerome Ng's avatar
Jerome Ng committed
154 155
  stage: manage
  group: compliance
156
  public: true
Jerome Ng's avatar
Jerome Ng committed
157
  pi_type: Other
158
  product_analytics_type: Both
159
  is_primary: false
160 161
  is_key: false
  health:
162
    level: 2
163
    reasons:
164 165
    - In Q3 we saw 936 new projects use compliance frameworks. These were a limited selection of 5, hard-coded options. We are targeting a 10% increase in adoption by making these frameworks [customizable](https://gitlab.com/gitlab-org/gitlab/-/issues/255340)
    - Additionally, we're using a conservative target for now while we focus on [several improvements](https://gitlab.com/groups/gitlab-org/-/epics/4082) to the underlying feature that drives this metric.
166 167 168 169
    - We're continuing iteration on [compliance frameworks](https://gitlab.com/groups/gitlab-org/-/epics/4082) to give them more utility and solve several standardization and enforcement problems for customers. 
    - We're adding an improvement to give organizations [better control](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) of who can add or modify these frameworks at the project level in `13.8`.
    - We expect the next iteration, consisting of [frontend](https://gitlab.com/gitlab-org/gitlab/-/issues/292696) and [backend](https://gitlab.com/gitlab-org/gitlab/-/issues/254389) additions to ship in `13.9`. This iteration is particularly important for the reason below.
    - By `13.10` we expect compliance frameworks to fully support one of our most requested solutions, [group compliance pipelines](https://gitlab.com/groups/gitlab-org/-/epics/3156), as a required step to implement it, which we expect to drive even more adoption of compliance frameworks.
Jerome Ng's avatar
Jerome Ng committed
170
  implementation:
171
    status: Complete
172
    reasons:
173
  metric_name: projects_with_compliance_framework
174
  sisense_data:
Matt Gonzales (ex-GitLab)'s avatar
Matt Gonzales (ex-GitLab) committed
175
    chart: 9071082
176 177
    dashboard: 663045
    embed: v2
178

179 180
- name: Manage:Import - GMAU - MAU importing
  base_path: "/handbook/product/performance-indicators/"
181
  definition: A monthly count of the unique number of users performing an import
182
  target: TBD - See Health section below for details
183
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
184
  section: dev
Jerome Ng's avatar
Jerome Ng committed
185 186
  stage: manage
  group: import
187
  public: true
Jerome Ng's avatar
Jerome Ng committed
188
  pi_type: GMAU
189
  product_analytics_type: Both
190
  is_primary: true
191 192 193 194
  is_key: false
  health:
    level: 0
    reasons:
195
    - No target defined yet.
196 197
    - Target will be defined after the [metric instrumentation](https://gitlab.com/gitlab-org/gitlab/-/issues/283175) is implemented and we get a baseline of data.
    - Improvement - Reducing friction for GitLab-to-GitLab migrations - The [MVC for the Group Migration feature](https://gitlab.com/groups/gitlab-org/-/epics/4374) was enabled in production in `13.9`. This feature will enable a much easier migration of an entire GitLab group from one instance to another. 
Jerome Ng's avatar
Jerome Ng committed
198
  implementation:
199
    status: Dashboard
200
    reasons:
201 202 203
    - Instrumentation has been [implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/283175) in `13.9`. 
    - Dashboard [update](https://gitlab.com/gitlab-data/analytics/-/issues/4244) is in progress. 
  metric_name: unique_users_all_imports
204
  
205 206 207 208
- name: Manage:Import - Other - Share of New Projects Imported
  base_path: "/handbook/product/performance-indicators/"
  definition: Share of projects created via import within 90 days of the namespace's
    creation.
209
  target: TBD - See Health section below for details
210
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
211
  section: dev
Jerome Ng's avatar
Jerome Ng committed
212 213
  stage: manage
  group: import
214
  public: true
Jerome Ng's avatar
Jerome Ng committed
215
  pi_type: Other
216
  product_analytics_type: Both
217 218 219 220 221
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
222
    - No target defined yet.
223
    - Target will be defined once the dashboard is [updated](https://gitlab.com/gitlab-data/analytics/-/issues/4244) with the improved metric (for both Saas and self-managed).
224
    - Improvement - Reducing friction for GitLab-to-GitLab migrations - The [MVC for the Group Migration feature](https://gitlab.com/groups/gitlab-org/-/epics/4374) was enabled in production in `13.9`. This feature will enable a much easier migration of an entire GitLab group from one instance to another.
Jerome Ng's avatar
Jerome Ng committed
225
  implementation:
Haris Delalić's avatar
Haris Delalić committed
226
    status: Dashboard
227
    reasons:
228 229
    - Instrumented for [both GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/224758) in `13.6`.
    - Dashboard [update](https://gitlab.com/gitlab-data/analytics/-/issues/4244) is currently in progress.
230
  metric_name: TBD 
Jerome Ng's avatar
Jerome Ng committed
231

232
- name: Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
233 234
  base_path: "/handbook/product/performance-indicators/"
  definition: Number of unique users interacting with an issue (open, close, reopen,
235
    adjust label, adjust milestone, adjust weight, comment) 
236
  target: 484,510 for All, 148,176 for Paid 
237
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
238
  section: dev
Jerome Ng's avatar
Jerome Ng committed
239
  stage: plan
240
  group: project_management
241
  public: true
242
  pi_type: SMAU, Paid GMAU
243
  product_analytics_type: Both
244 245
  is_primary: true
  is_key: false
246 247
  lessons:
    learned:
248
      - Only 28% of instances are currently on a version with the new Plan xMAU metric counters. 
249
      - The ratio of [net new Paid GMAU to churned Paid GMAU](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)?widget=10634701&udv=0) improved from 3.75 in December to 3.99 in January, which indicates net growth and retention for Plan Paid users is healthy.
250
      - MAU interacting with Iterations increased by 52% from December to January. The next step to drive feature adoption is more holistically integrating Iterations into Boards. 
251 252 253
  monthly_focus:
    goals: 
      - Driver - Determine technical feasibility for allowing an [issue to be assigned to multiple groups](https://gitlab.com/gitlab-org/gitlab/-/issues/296668), which would likely solve [Plan adoption blockers (priorities 1 and 3)](https://gitlab.com/gitlab-org/gitlab/-/issues/218333).
254
      - Continue making progress on the [13.9 release plan](https://gitlab.com/gitlab-org/plan/-/issues/236) and roll into [13.10 goals](https://gitlab.com/gitlab-org/plan/-/issues/264).
255
      - Open > 3 documentation MRs in support of [better articulating Plan use cases](https://gitlab.com/gitlab-org/gitlab/-/issues/213810).
256
      - Complete the validation track for [saved filters/searches](https://gitlab.com/gitlab-org/gitlab/-/issues/14512), [customizable issue types](https://gitlab.com/gitlab-org/gitlab/-/issues/118666), and [interacting with a full issue from within a Board](https://gitlab.com/gitlab-org/gitlab/-/issues/13319)
257
      - Merge architectural blueprint for [consolidating Groups and Projects](https://gitlab.com/gitlab-org/architecture/tasks/-/issues/7) on the backend.
258 259 260
  health:
    level: 3
    reasons:
261
    - Achieved - FY21 Q4 targets of 437,490 for All, 101,835 for Paid. New targets set for FY22 Q1. 
262 263
    - Driver - For Paid, continue high level progress on [achieving feature parity between an Instance, Group, and Project](https://gitlab.com/groups/gitlab-org/-/epics/2885), [making issues more extensible](https://gitlab.com/groups/gitlab-org/-/epics/3354), and continuing to make incremental progress on [iterations](https://gitlab.com/groups/gitlab-org/-/epics/2422).
    - Improvement - For All, work towards addressing the [problem areas](https://gitlab.com/groups/gitlab-org/-/epics/4104) that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
264 265
    - Improvement - For All, continue making progress on [quality of life improvements](https://gitlab.com/groups/gitlab-org/-/epics/3216) for existing features. 
    - Improvement - For Paid, creating multi-level Epics spanning a Group hierarchy is clunky and non-intuitive. Improving this, along with making the Roadmap more interactive and intuitive would be a huge catalyst to drive Product Planning adoption. 
266
    - Improvement - For Paid, continue making incremental, iterative progress towards [Plan's core JTBD](/direction/plan/#problem-and-jobs-to-be-done) as outlined in the [Issue Tracking direction](/direction/plan/issue_tracking/). 
267 268
  implementation:
    status: Complete
269
    reasons:
270 271
    - xMAU data models are now updated with the new counter that aggregates unique users across many different issue actions.
    - New [Plan xMAU](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)) dashboard is live.
272
  metric_name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly 
273
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
274 275
    chart: 9967544
    dashboard: 758607
Jerome Ng's avatar
Jerome Ng committed
276 277
    embed: v2
  sisense_data_secondary:
Jerome Ng's avatar
Jerome Ng committed
278 279
    chart: 9980295
    dashboard: 758607
280
    embed: v2
281
  urls:
282
  - https://about.gitlab.com/direction/plan/issue_tracking/
283
  - https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)
Jerome Ng's avatar
Jerome Ng committed
284

Jerome Ng's avatar
Jerome Ng committed
285
- name: Plan:Product Planning - Paid GMAU - MAU creating Epics each Month
286 287
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users creating Epics each month.
288
  target: 3,900 Combined Unique Users for Self Managed and SaaS
289
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
290
  section: dev
Jerome Ng's avatar
Jerome Ng committed
291
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
292
  group: product_planning
293
  public: true
Jerome Ng's avatar
Jerome Ng committed
294
  pi_type: Paid GMAU
295
  product_analytics_type: Both
296 297 298 299 300
  is_primary: true
  is_key: false
  health:
    level: 3
    reasons:
Christen Dybenko's avatar
Christen Dybenko committed
301 302
    - Achieved - Target of 3,900 (3,943 actual GMAU for epic creation) 
    - Insight - [Epic creation](https://app.periscopedata.com/app/gitlab/602355/Portfolio-Management-G-SMAU-and-Feature-Success-Tracking?widget=9245316&udv=962108) for SaaS continues to grow. Self-Managed was stable, but that could be due to the [Janauary tracking bug](https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/290).
303
    - Improvement - [Epic Boards](https://gitlab.com/groups/gitlab-org/-/epics/2864) MVC 1 is planned to ship in 13.10 and should continue the trend into the end of FY21 by enabling Portfolio-level planning.  
304
    - Improvement - [Epic Swimlanes](https://gitlab.com/groups/gitlab-org/-/epics/3352) shipped in 13.6
Christen Dybenko's avatar
Christen Dybenko committed
305
    - Closed - original [PI dashboard issue for epic creation](https://gitlab.com/gitlab-data/analytics/-/issues/5280#note_505497453) 
Jerome Ng's avatar
Jerome Ng committed
306 307
  implementation:
    status: Complete
308
    reasons:
Christen Dybenko's avatar
Christen Dybenko committed
309 310 311
    - We consider our Epic Creation GMAU as "Complete". However, our next target will be set after a second instrumentation.
    - Instrument "all epic interactions" during 13.9 - 13.10.  [Instrumentation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/292253) and [dasboard issue](https://gitlab.com/gitlab-data/analytics/-/issues/7885).
    - A new target will be selected when we have a baseline.
312
  metric_name: epics 
313
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
314 315
    chart: 9980303
    dashboard: 758607
316 317
    embed: v2
  urls:
Christen Dybenko's avatar
Christen Dybenko committed
318
    
Mark Wood's avatar
Mark Wood committed
319
- name: Plan:Certify - Paid GMAU - MAU Interacting with Requirements
320 321 322 323 324
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users interacting (create, view, reference) each
    month
  target: TBD
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
325
  section: dev
Jerome Ng's avatar
Jerome Ng committed
326
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
327
  group: product_planning
328
  public: true
Jerome Ng's avatar
Jerome Ng committed
329
  pi_type: Paid GMAU
330
  product_analytics_type: Both
Jerome Ng's avatar
Jerome Ng committed
331
  is_primary: false
332 333
  is_key: false
  health:
Jerome Ng's avatar
Jerome Ng committed
334
    level: 0
335
    reasons:
Mark Wood's avatar
Mark Wood committed
336 337 338
    - Note - [Requirement interaction tracking](https://gitlab.com/gitlab-org/product-intelligence/-/issues/370) issue is complete. 
      Data is in the process of being aggregated. Early data shows healthy adoption of test report linking.
      [Tracking issue](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
339
    - Improvement - UX improvements for [Requirements Management](https://gitlab.com/groups/gitlab-org/-/epics/3708)
Mark Wood's avatar
Mark Wood committed
340
      based on completed research study, Requirement [Export MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/290813).
Jerome Ng's avatar
Jerome Ng committed
341 342
  implementation:
    status: Dashboard
343
    reasons:
Justin Farris's avatar
Justin Farris committed
344
    - Dashboarding - Requirement Creation is instrumented is complete. Tracking issue for gmau dashboard [here](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
345
  metric_name: TBD
Mark Wood's avatar
Mark Wood committed
346
- name: Plan:Certify - Other - Number of Requirements Created Monthly
347 348
  base_path: "/handbook/product/performance-indicators/"
  definition: A monthly count of the number of requirements created.
Mark Wood's avatar
Mark Wood committed
349
  target: 1000 by end of March 2021
350
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
351
  section: dev
Jerome Ng's avatar
Jerome Ng committed
352
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
353
  group: product_planning
354
  public: true
Jerome Ng's avatar
Jerome Ng committed
355
  pi_type: Other
356
  product_analytics_type: Both
357 358 359 360 361
  is_primary: false
  is_key: false
  health:
    level: 3
    reasons:
Mark Wood's avatar
Mark Wood committed
362 363
    - Improvement - [Requirement Export MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/290813) feature to release in 13.8
    - Insight - Requirement creation has decreased. Investigation shows a substantial set of users are Universities, which explains the downturn for the holidays. 
Mark Wood's avatar
Mark Wood committed
364
    - Insight - Self-managed instances creating requirements continues to grow.   
Jerome Ng's avatar
Jerome Ng committed
365 366
  implementation:
    status: Dashboard
367
    reasons:
368
    - Dashboarding -  accurate instrumentation on Self Managed, still needs verification
369
      for SaaS - [issue](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
Jerome Ng's avatar
Jerome Ng committed
370
  metric_name: requirements_created
371
  sisense_data:
Mark Wood's avatar
Mark Wood committed
372
    chart: 9666808
373
    dashboard: 634539
374 375 376
    embed: v2
  urls:
  - https://gitlab.com/gitlab-data/analytics/-/issues/5270
377

James Ramsay's avatar
James Ramsay committed
378
- name: Create - SMAU - Unique users writing to a Git repository
379
  base_path: "/handbook/product/performance-indicators/"
380 381
  definition: A rolling count of the number of unique users who performed any Git write operation within the last 28 days.
  target: |
James Ramsay's avatar
James Ramsay committed
382
    Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
383
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
384
  section: dev
Jerome Ng's avatar
Jerome Ng committed
385
  stage: Create
386
  public: true
Jerome Ng's avatar
Jerome Ng committed
387
  pi_type: SMAU
388
  product_analytics_type: Both
389 390 391
  is_primary: true
  is_key: false
  health:
392
    level: 3
393
    reasons:
James Ramsay's avatar
James Ramsay committed
394 395 396 397 398
    - Observation - December 2020 SMAU is less than November 2020 SMAU. A
      similar drop can be observed in GMAU as well, suggesting a common cause.
      The cause is likely seasonality consistent with the end of year. For this
      reason January SMAU and GMAU is expected to return to November 2020
      levels and continue growing.
James Ramsay's avatar
James Ramsay committed
399 400 401 402 403 404
    - Insight - on the median GitLab instance, 33% of users perform a Git write
      operation each month
      ([source](https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=10435358&udv=1100338)).
    - Insight - on the median Paid GitLab instance (self-managed), 50% of users
      perform a Git write operation each month
      ([source](https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=10435361&udv=1100338)).
Jerome Ng's avatar
Jerome Ng committed
405 406
  implementation:
    status: Complete
407
    reasons:
James Ramsay's avatar
James Ramsay committed
408 409 410 411
    - Improvement - AMAU is currently over counted. A deduplicated counter was implemented
      [gitlab#234027](https://gitlab.com/gitlab-org/gitlab/-/issues/234027) in GitLab 13.6.
      The charts do not yet reflect the new counter
      [gitlab#291943](https://gitlab.com/gitlab-org/gitlab/-/issues/291943).
412 413
  metric_name: action_monthly_active_users_project_repo
  sisense_data:
Kathleen Tam's avatar
Kathleen Tam committed
414 415
    chart: 9967642
    dashboard: 758607
416
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
417 418 419 420
  sisense_data_secondary:
    chart: 9980442
    dashboard: 758607
    embed: v2
421
  urls:
Daniel Gruesso's avatar
Daniel Gruesso committed
422
  - https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard
423

James Ramsay's avatar
James Ramsay committed
424
- name: Create:Source Code - GMAU - Unique users writing to a project Git repository
425
  base_path: "/handbook/product/performance-indicators/"
426 427 428
  definition: A rolling count of the number of unique users who performed any Git write
    operation within the last 28 days
  target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
429
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
430
  section: dev
Jerome Ng's avatar
Jerome Ng committed
431 432
  stage: create
  group: source_code
433
  public: true
Jerome Ng's avatar
Jerome Ng committed
434
  pi_type: GMAU
435
  product_analytics_type: Both
436 437 438
  is_primary: true
  is_key: false
  health:
Daniel Gruesso's avatar
Daniel Gruesso committed
439
    level: 3
440
    reasons:
441
    - Starting in [GitLab 13.6](https://gitlab.com/gitlab-org/gitlab/-/issues/234027)
442
      AMAU for this metric takes into consideration all repository types.
443 444 445 446
    - Improvement - work towards providing users more insights while navigating their
      code with [Code Intelligence](https://gitlab.com/groups/gitlab-org/-/epics/1576)
    - Improvement - work towards providing more control over [managing repository
      size](https://gitlab.com/groups/gitlab-org/-/epics/3793)
Jerome Ng's avatar
Jerome Ng committed
447
  implementation:
448
    status: Complete
449
    reasons:
450
    - Enabled on GitLab.com
451 452
    - Released for self-managed in GitLab 13.3
  metric_name: action_monthly_active_users_project_repo
Daniel Gruesso's avatar
Daniel Gruesso committed
453
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
454 455 456 457 458 459
    chart: 9967642
    dashboard: 758607
    embed: v2
  sisense_data_secondary:
    chart: 9980442
    dashboard: 758607
Daniel Gruesso's avatar
Daniel Gruesso committed
460
    embed: v2
461
  urls:
Daniel Gruesso's avatar
Daniel Gruesso committed
462
  - https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=9520571&udv=0
James Ramsay's avatar
James Ramsay committed
463
  - https://gitlab.com/gitlab-org/gitlab/-/issues/234027
464

465
- name: Create:Source Code Secondary - GMAU - Unique users approving merge requests
466
  base_path: "/handbook/product/performance-indicators/"
467
  definition: A rolling count of the number of unique users who approved any merge request within the last 28 days
468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: source_code
  public: true
  pi_type: GMAU
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Not currently instrumented
    - Currently planning instrumentation and evaluating if this is a more relevant metric for the Source Code group
  urls:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/293014

485
- name: Create:Source Code Tertiary - GMAU - Unique users pushes rejected by push rules
486
  base_path: "/handbook/product/performance-indicators/"
487
  definition: A rolling count of the number of unique users who had one or more push rejected by any push rule within the last 28 days
488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: source_code
  public: true
  pi_type: GMAU
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Not currently instrumented
    - Currently planning instrumentation and evaluating if this is a more relevant metric for the Source Code group
  urls:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/293016

James Ramsay's avatar
James Ramsay committed
505
- name: Create:Code Review - GMAU - Unique users using Merge Requests
506
  base_path: "/handbook/product/performance-indicators/"
507 508
  definition: This is defined as users who perform one or more of the following actions in
   a merge request - create, merge, or close a Merge Request.
Kai Armstrong's avatar
Kai Armstrong committed
509
  target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU. 
510
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
511
  section: dev
Jerome Ng's avatar
Jerome Ng committed
512
  stage: create
513
  group: code_review
514
  public: true
James Ramsay's avatar
James Ramsay committed
515
  pi_type: GMAU
516
  product_analytics_type: Both
517
  is_primary: true
518 519
  is_key: false
  health:
520
    level: 3
521
    reasons:
Kai Armstrong's avatar
Kai Armstrong committed
522
    - Target - This aligns with our current understanding that Code Review is about 50% of the MAU of the stage metric and continuing to maintain that difference.
523
    - Insight - Paid users of Code Review make up about one-third of total Code Review users - this provides an opportunity to convert to paid and increase users.
524
    - Improvement - Support for [custom commit messages when applying a suggested change](https://gitlab.com/gitlab-org/gitlab/-/issues/25381) will allow more teams with strict commit message guidelines to adopt this feature and maintain conformity with the project.
525 526
    - Improvement - [Mark changes as viewed](https://gitlab.com/gitlab-org/gitlab/-/issues/17531/) in merge requests is a foundational step towards tracking and communicating progress on merge requests to other participants.
    - Improvement - SCM <-> Code Review <-> Verify correlation charts now defined in [new charts](#createcode-review---other---code-reviewscm-ratio-by-umau-and-subscription-tier)
Jerome Ng's avatar
Jerome Ng committed
527
  implementation:
528
    status: Complete
529
    reasons:
530 531
    - Enabled on GitLab.com 2020-07-04
    - Enabled for self-managed on GitLab 13.2
532
    - Working on more comprehensive MAU tracking in [&4948](https://gitlab.com/groups/gitlab-org/-/epics/4948)
533
  metric_name: merge_requests_users 
Daniel Gruesso's avatar
Daniel Gruesso committed
534
  sisense_data:
535 536
    chart: 10426645
    dashboard: 786738
Daniel Gruesso's avatar
Daniel Gruesso committed
537
    embed: v2
538
  sisense_data_secondary:
539 540
    chart: 10624715
    dashboard: 786738
541
    embed: v2
542
  urls:
543
  - https://app.periscopedata.com/app/gitlab/526749/Create---Code-Review-category-dashboard
James Ramsay's avatar
James Ramsay committed
544

545
- name: Create:Code Review - Other - Code Review/SCM Ratio by UMAU and Subscription Tier
Kai Armstrong's avatar
Kai Armstrong committed
546 547
  base_page: "/handbook/product/performance-indicators/"
  definition: 
548
  target: TBD - Primarily this metric is used for monitoring the relationship between SCM GMAU and Code Review GMAU to make sure it doesn't not fall out of alignment.
Kai Armstrong's avatar
Kai Armstrong committed
549 550 551 552 553 554 555 556 557 558 559 560 561
  org: Dev Section
  section: dev
  stage: create
  group: code_review
  public: true
  pi_type: Other
  product_analytics_type: Both
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Target - this metric is primarily monitored to remain at a consistent level.
562 563
    - Insight - Paid Instances with 10+ users more closely track to 75% of SCM Users using Code Review
    - Insight - Free Instances and those with less than 10 users drive down the SCM<->Code Review Ratio as most have less than 50% adoption
Kai Armstrong's avatar
Kai Armstrong committed
564 565 566 567
  implementation:
    status: Complete
  metric_name: merge_requests_users
  sisense_data:
568 569 570 571 572 573
    chart: 10728688
    dashboard: 801102
    embed: v2
  sisense_data_secondary:
    chart: 10688270
    dashboard: 801102
Kai Armstrong's avatar
Kai Armstrong committed
574
    embed: v2
575 576
  urls:
  - https://app.periscopedata.com/app/gitlab/801102/SCM-%3C%3E-Code-Review-%3C%3E-Verify
577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601
  - https://app.periscopedata.com/app/gitlab/801102/SCM-%3C%3E-Code-Review-%3C%3E-Verify?widget=10728688&udv=1218043

- name: Create:Code Review - Editor Extension (VS Code) - Category MAU
  base_path: "/handbook/product/performance-indicators/"
  definition: This is defined as users who use [GitLab Workflow](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow) for VS Code and make at least one API request to a GitLab instance.
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: code_review
  public: true
  pi_type: Other
  product_analytics_type: Both
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Instrumentation in 13.9
    - Improvement - [GitLab Workflow 3.11.0](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CHANGELOG.md#3110-2021-01-25) released with auto completion for GitLab CI Variables
    - Improvement - [GitLab Workflow 3.10.0](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CHANGELOG.md#3100-2021-01-19) released with viewing code review comments on merge request diffs
  implementation:
    status: Instrumentation
    reasons:
    - MAU Collection [Issue](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/issues/258)
Kai Armstrong's avatar
Kai Armstrong committed
602

603
- name: Create:Gitaly - Other - Availability
604
  base_path: "/handbook/product/performance-indicators/"
605 606 607 608 609 610 611
  definition: Percentage of time during which Gitaly is fully operational and providing
    service to users within SLO parameters. Availability is measured according by
    Apdex (latency) and error rate. Gitaly is considered available when at least 50%
    of users are experiencing satisfactory latency, _and_ 50% of requests are completing
    successfully. If either of these conditions is not met, the service is experiencing
    an outage.
  target: 99.95%, consistent with GitLab.com availability service level objectives
612
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
613
  section: dev
614 615
  stage: create
  group: gitaly
616
  public: true
Jeremy Watson (ex-GitLab)'s avatar
Jeremy Watson (ex-GitLab) committed
617
  pi_type: Other
618
  product_analytics_type: Both
619 620 621 622 623 624 625 626 627 628
  is_primary: true
  is_key: false
  health:
    level: 0
    reasons:
    - Instrumentation for this metric started in 13.4 and is still ongoing.
  implementation:
    status: Instrumentation
    reasons:
    - Rolling 7 day metrics available in [Grafana](https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview)
Mark Wood's avatar
Mark Wood committed
629
      for GitLab.com. Current availability per Adpex is 99.97%.
630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646
    - Improvement - self-managed availability metrics are [planned](https://gitlab.com/gitlab-org/gitlab/-/issues/216683)
  metric_name: gitaly_apdex
  urls:
  - https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
  - https://gitlab.com/gitlab-org/gitlab/-/issues/216683

- name: Create:Gitaly - Other - Unique instances with Gitaly Cluster enabled
  base_path: "/handbook/product/performance-indicators/"
  definition: Number of recorded GitLab instances with Gitaly Cluster enabled
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: gitaly
  public: true
  pi_type: Other
  product_analytics_type: Self-Managed
Christen Dybenko's avatar
Christen Dybenko committed
647
  is_primary: false
648 649
  is_key: false
  health:
650
    level: 2
651
    reasons:
652 653
    - No target defined as the feature is still being rolled out.
    - Improvement - introduce [Variable Replication Factor and auto-reblancing within a cluster](https://gitlab.com/groups/gitlab-org/-/epics/3372) to enable more enterprise-level adoption.
Jerome Ng's avatar
Jerome Ng committed
654
  implementation:
655
    status: Complete
656
    reasons:
Mark Wood's avatar
Mark Wood committed
657
    - Insight - Seeing impressive growth in Gitaly Cluster adoption.
658 659
    - Instrumentation for reporting on instances with Gitaly Cluster enabled was introduced in 13.0 with the GA of Gitaly Cluster.
  metric_name: gitaly_clusters
Jerome Ng's avatar
Jerome Ng committed
660
  sisense_data:
661 662
    chart: 9504807
    dashboard: 728407
Jerome Ng's avatar
Jerome Ng committed
663
    embed: v2
664
  urls:
665
  - https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
James Ramsay's avatar
James Ramsay committed
666 667

- name: Create:Editor - GMAU - Unique users committing via any web editor
668 669
  base_path: "/handbook/product/performance-indicators/"
  definition: A rolling count of the number of unique users who performed a Git write
670 671
    operation to any respository using the Web IDE, Web Editor or Snippet within a
    stage in the last 28 days
Eric Schurter's avatar
Eric Schurter committed
672
  target: 4% growth month over month for All 
673
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
674
  section: dev
Jerome Ng's avatar
Jerome Ng committed
675 676
  stage: create
  group: editor
677
  public: true
Jerome Ng's avatar
Jerome Ng committed
678
  pi_type: GMAU
679
  product_analytics_type: Both
680
  is_primary: true
681 682
  is_key: false
  health:
683
    level: 2
684
    reasons:
685
    - New target GMAU to be set once we get cumulative, de-duped data from newly aligned categories.
686 687
    - Insight - Rebound from the holiday season to nearly November levels. We'll be monitoring to ensure consistent organic growth in subsequent months. 
    - Improvement - No targeted efforts in the next few releases to specifically grow GMAU, but currently working to [migrate the Web IDE to use Editor Lite](https://gitlab.com/gitlab-org/gitlab/-/issues/297449) which will bring consistency across editors and a [moderate performance improvement](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51212#note_483338464). 
Jerome Ng's avatar
Jerome Ng committed
688
  implementation:
Jerome Ng's avatar
Jerome Ng committed
689
    status: Complete
690
    reasons:
691
    - Instrumentation of [Static Site Editor and Wiki to roll up into GMAU](https://gitlab.com/gitlab-org/gitlab/-/issues/283947) completed in 13.7, waiting for data to be reflected in the charts.
692
  metric_name: action_monthly_active_users_ide_edit 
693
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
694 695
    chart: 9967737
    dashboard: 758607
696
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
697 698 699
  sisense_data_secondary:
    chart: 9980340
    dashboard: 758607
Jerome Ng's avatar
Jerome Ng committed
700
    embed: v2
701 702
  urls:
  - https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU
James Ramsay's avatar
James Ramsay committed