dev_section.yml 47.8 KB
Newer Older
Eric Brinkman's avatar
Eric Brinkman committed
1
- name: Dev - Section TMAU - Estimated & Recorded Sum of Dev Section SMAU
2 3
  base_path: "/handbook/product/performance-indicators/"
  definition: A sum of all SMAUs across the three stages in the Dev Section (Manage,
Jerome Ng's avatar
Jerome Ng committed
4
    Plan, and Create) for All users and Paid users.
Eric Brinkman's avatar
Eric Brinkman committed
5
  target: Estimated TMAU growth 4% MoM; 2.73M by end of FY21Q4 for TMAU, 603K by end of FY21Q4 for Paid TMAU
6
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
7
  section: dev
8
  public: true
Jerome Ng's avatar
Jerome Ng committed
9
  pi_type: Section TMAU
10
  product_analytics_type: Both
11 12 13
  is_primary: true
  is_key: false
  health:
14
    level: 3
15
    reasons:
Eric Brinkman's avatar
Eric Brinkman committed
16 17 18
    - We saw a 3.5% drop in Dev Section TMAU between Nov 2020 and Dec 2020. UMAU dropped 5.2% as a reference point.
    - Despite the drop, the Dev section has already achieved our end of Q4 target of 2.75M TMAU by 300K users.
    - TMAU drops were the largest for Create. Plan declined slightly by 3K users and Manage increased by 8K users, most likely because of upgrades as the GMAU was only turned on by default in 13.5.
19
    - Self-managed TMAU increased, but SaaS significantly decreased, likely to free users declining due to seasonality - as a next step, we'll work to add a 4th graph in this section with TMAU broken out by tiers, including free and paid.
Eric Brinkman's avatar
Eric Brinkman committed
20 21
    - Insight - instances with large amounts of users are unlikely to upgrade immediately, typically taking a few months to do so. So, our larger instances were likely counted as estimated earlier and as the larger instances upgrade, what we'll likely see is that estimated uplift was under reported early. It's difficult to tell if it will be over reported as our largest self-managed instances are unlikely to ever report back. It will be interesting to see how the uplifts change when we base the uplift on users instead of subscriptions.
    - Improvement - For All Usage, address the [problems](https://gitlab.com/groups/gitlab-org/-/epics/4104)
22
      that emerged out of the Category Maturity Scorecard exercise for Issue Tracking
Eric Brinkman's avatar
Eric Brinkman committed
23
      and continue making progress on [usability improvements](https://gitlab.com/groups/gitlab-org/-/epics/3216)
24
      for existing Plan features.
Eric Brinkman's avatar
Eric Brinkman committed
25 26 27 28 29 30
    - Improvement - For All Usage, Merge Request [usability improvements](https://gitlab.com/groups/gitlab-org/-/epics/3607)
    - Improvement - For All Usage, [MR reviewers](https://gitlab.com/gitlab-org/gitlab/-/issues/237921) will make us more competitive with GitHub and should help with the lovability of assigning merge requests
    - Improvement - For All Usage, [Iterations in GitLab](https://gitlab.com/groups/gitlab-org/-/epics/2422) will make plan more usable for many Agile development teams
    - Improvement - For Paid Usage, [Epic Boards](https://gitlab.com/groups/gitlab-org/-/epics/4117) will allow for much better portfolio management via boards
    - Improvement - For Paid Usage, [SSO Group Sync](https://gitlab.com/gitlab-org/gitlab/-/issues/118)
    - Improvement - For All Usage, [Workspaces](https://gitlab.com/groups/gitlab-org/-/epics/4257)
Eric Brinkman's avatar
Eric Brinkman committed
31 32
    - Improvement - For All Usage, [Groups/Project Simplification](https://gitlab.com/groups/gitlab-org/-/epics/2885)
    - Improvement - For All Usage, [Settings and Navigation improvements](https://gitlab.com/groups/gitlab-org/-/epics/4831)
Jerome Ng's avatar
Jerome Ng committed
33
  implementation:
34
    status: Complete
Eric Brinkman's avatar
Eric Brinkman committed
35
    reasons:
36
  metric_name: issues, action_monthly_active_users_project_repo, analytics_unique_visits_for_any_target
Eric Brinkman's avatar
Eric Brinkman committed
37
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
38 39
    chart: 9982128
    dashboard: 758607
40 41
    embed: v2
  sisense_data_secondary:
42 43
    chart: 10138877
    dashboard: 758607
Eric Brinkman's avatar
Eric Brinkman committed
44
    embed: v2
Mathieu Peychet's avatar
Mathieu Peychet committed
45 46 47 48
  sisense_data_tertiary:
    chart: 9992267
    dashboard: 758607
    embed: v2
49

Jeremy Watson (ex-GitLab)'s avatar
Jeremy Watson (ex-GitLab) committed
50
- name: Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features
51
  base_path: "/handbook/product/performance-indicators/"
52
  definition: The number of unique users viewing any <a href="https://docs.gitlab.com/ee/user/analytics/">Analytics page</a>.
53
  target: 116,050
54
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
55
  section: dev
Jerome Ng's avatar
Jerome Ng committed
56
  stage: manage
57
  group: analytics
58
  public: true
59
  pi_type: SMAU, GMAU
60
  product_analytics_type: Both
61
  is_primary: true
62 63
  is_key: false
  health:
64
    level: 3
65
    reasons:
66
    - Insights - The Optimize data for January is inaccurate due to [this usage ping issue](https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/290). Many page views for self managed are showing as negative.
67 68 69 70
    - Insights - Optimize feature usage is now above target. 
    - Insights - SaaS usage of Optimize features dropped 10% in December. The most significant drop was a 13.7% decrease in views of CI/CD analytics by SaaS users. In December and previous months this page would often give a 500 error. This looks like it has been resolved now.
    - Insights - Self-managed usage of Optimize features surpassed SaaS usage for the first time in December. This was due to a combination of a decline in usage by SaaS users and an increase in usage by self-managed users.
    - Insights - Usage by self-managed users increased 23% in December. We have been focused on improving instance-level analytics over the past 3 milestones and the data shows that these features are being adopted. In fact, most of the increase in self-managed usage is the result of interest in instance-level analytics. DevOps Reports usage increased by 35% in December. Instance-level Usage Trends was announced in 13.6 and is gaining popularity. DevOps Score and Usage Trends combined accounted for 50% of the increase in usage of Optimize features on self managed. Instance-level analytics are still only available to Admin users on self-managed instances. As we make these available to SaaS users in 2021 and continue to iterate on them, we expect to see further adoption and an overall improvement in the Optimize PI.
71 72
    - Improvements - DevOps Adoption encountered [a setback](https://gitlab.com/gitlab-org/gitlab/-/issues/296972) that has delayed its rollout. There has been interest in this feature from enterprise customers wanting insights into how GitLab is being used in their organizations. We are investigating the best approach to moving forward on this feature.
    - SaaS data is not available for the paid SMAU chart due to [limitations of the redis counter that is used](https://gitlab.com/gitlab-data/analytics/-/issues/6603#note_484785813).
Jerome Ng's avatar
Jerome Ng committed
73
  implementation:
74
    status: Complete
75
  metric_name: analytics_total_unique_counts_monthly
76
  sisense_data:
Kathleen Tam's avatar
Kathleen Tam committed
77 78
    chart: 10039390
    dashboard: 758607
79
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
80 81 82 83 84
  sisense_data_secondary:
    chart: 10039413
    dashboard: 758607
    embed: v2

85 86 87
- name: Manage:Access - Paid GMAU - MAU Using Paid SAML
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users logging into GitLab using SAML SSO for Groups.
88
  target: 30000. See https://gitlab.com/gitlab-com/Product/-/issues/1832 for further detail.
89
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
90
  section: dev
Jerome Ng's avatar
Jerome Ng committed
91 92
  stage: manage
  group: access
93
  public: true
Jerome Ng's avatar
Jerome Ng committed
94
  pi_type: Paid GMAU
Melissa Ushakov's avatar
Melissa Ushakov committed
95
  product_analytics_type: SaaS
96 97 98
  is_primary: true
  is_key: false
  health:
99
    level: 3
100
    reasons:
101
    - Improvements - In 13.10, Auditor users will have [read-only access to groups with SSO enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/300455). This will allow the support organization to reduce the number of Administrators for GitLab.com and meet their [OKR](https://gitlab.com/gitlab-com/support/managers/okrs/-/issues/29)! 
102 103 104
    - Improvements - In 13.10, GitLab will begin [enforcing SSO for Git activity](https://gitlab.com/gitlab-org/gitlab/-/issues/22754). This will provide the same level of protection to Git as the Web UI and been requested by several large customers and prospects. 
    - Improvements - In 13.10, we will increase security for GitLab.com by [lowering the timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/298756) for SAML SSO sessions from 7 days to 1 day. 
    - Insights - We continue to see a healthy upward trend for group SAML logins per month! This is a combination of decreasing the SSO timeout and enteprise customer growth for GitLab.com . 
Jerome Ng's avatar
Jerome Ng committed
105
  implementation:
Kathleen Tam's avatar
Kathleen Tam committed
106
    status: Complete 
107
    reasons:
Melissa Ushakov's avatar
Melissa Ushakov committed
108
  metric_name: user_auth_by_provider (Group_SAML)
Kathleen Tam's avatar
Kathleen Tam committed
109 110 111 112
  sisense_data:
    chart: 10039419
    dashboard: 758607
    embed: v2
113 114 115 116
- name: Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users viewing Audit Events, Compliance Dashboard,
    or Credential Inventory.
117
  target: TBD (expected [April 2021](https://gitlab.com/gitlab-org/manage/compliance/general/-/issues/13))
118
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
119
  section: dev
Jerome Ng's avatar
Jerome Ng committed
120 121
  stage: manage
  group: compliance
122
  public: true
Jerome Ng's avatar
Jerome Ng committed
123
  pi_type: Paid GMAU
124
  product_analytics_type: Both
125 126 127
  is_primary: true
  is_key: false
  health:
128
    level: 2
129
    reasons:
130 131
    - We're still [building out the compliance features](https://gitlab.com/gitlab-org/manage/compliance/general/-/issues/13) that are tracked by this metric. This improvement work will take longer than desired to complete since we do not have enough people to ship any faster. We expect iterations to ship on a continuous basis across each of these features.
    - Setting a target is premature since we're still collecting initial data. We would like to collect a total of 6 months worth of data before setting a target. 
132
    - Result - Increased GMAU by ~25% in December [by making Audit Events available to Developers and making the feature more discoverable](https://gitlab.com/gitlab-org/gitlab/-/issues/257514).
133 134 135
    - Adding a [Change Management view](https://gitlab.com/groups/gitlab-org/-/epics/4802) to the Compliance Dashboard. Planned for `Q2 2021`.
    - Improving PAT & SSH [credential management](https://gitlab.com/groups/gitlab-org/-/epics/3084). Ongoing with expected epic completion in `Q2 2021`.
    - Improving [discoverability](https://gitlab.com/groups/gitlab-org/-/epics/4441) of Compliance features. Planned for `Q2 2021`.
136
        - Adding [additional credentials](https://gitlab.com/groups/gitlab-org/-/epics/4110) to the credential inventory. Planned for `Q3 2021`.
Jerome Ng's avatar
Jerome Ng committed
137
  implementation:
138
    status: Complete 
139
    reasons:
140
  metric_name: i_compliance_audit_events, g_compliance_audit_events, g_compliance_dashboard, i_compliance_credential_inventory, usage_data_g_compliance_dashboard, usage_data_a_compliance_audit_events_api
141
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
142 143
    chart: 9980315
    dashboard: 758607
144
    embed: v2
145 146 147 148
  sisense_data_secondary:
    chart: 10039422
    dashboard: 758607
    embed: v2
149
    
150
- name: Manage:Compliance - Other - Projects Marked with a Compliance Framework
151
  base_path: "/handbook/product/performance-indicators/"
152
  definition: New and cumulative number of projects using compliance frameworks.
153
  target: 1,030 total new projects in Q4 using compliance frameworks (SaaS and self-managed combined)
154
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
155
  section: dev
Jerome Ng's avatar
Jerome Ng committed
156 157
  stage: manage
  group: compliance
158
  public: true
Jerome Ng's avatar
Jerome Ng committed
159
  pi_type: Other
160
  product_analytics_type: Both
161
  is_primary: false
162 163
  is_key: false
  health:
164
    level: 2
165
    reasons:
166 167
    - In Q3 we saw 936 new projects use compliance frameworks. These were a limited selection of 5, hard-coded options. We are targeting a 10% increase in adoption by making these frameworks [customizable](https://gitlab.com/gitlab-org/gitlab/-/issues/255340)
    - Additionally, we're using a conservative target for now while we focus on [several improvements](https://gitlab.com/groups/gitlab-org/-/epics/4082) to the underlying feature that drives this metric.
168 169 170 171
    - We're continuing iteration on [compliance frameworks](https://gitlab.com/groups/gitlab-org/-/epics/4082) to give them more utility and solve several standardization and enforcement problems for customers. 
    - We're adding an improvement to give organizations [better control](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) of who can add or modify these frameworks at the project level in `13.8`.
    - We expect the next iteration, consisting of [frontend](https://gitlab.com/gitlab-org/gitlab/-/issues/292696) and [backend](https://gitlab.com/gitlab-org/gitlab/-/issues/254389) additions to ship in `13.9`. This iteration is particularly important for the reason below.
    - By `13.10` we expect compliance frameworks to fully support one of our most requested solutions, [group compliance pipelines](https://gitlab.com/groups/gitlab-org/-/epics/3156), as a required step to implement it, which we expect to drive even more adoption of compliance frameworks.
Jerome Ng's avatar
Jerome Ng committed
172
  implementation:
173
    status: Complete
174
    reasons:
175
  metric_name: projects_with_compliance_framework
176
  sisense_data:
Matt Gonzales (ex-GitLab)'s avatar
Matt Gonzales (ex-GitLab) committed
177
    chart: 9071082
178 179
    dashboard: 663045
    embed: v2
180

181 182 183
- name: Manage:Import - GMAU - MAU importing
  base_path: "/handbook/product/performance-indicators/"
  definition: A monthly count of the unique number of users performing an import.
184
  target: TBD - See Health section below for details
185
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
186
  section: dev
Jerome Ng's avatar
Jerome Ng committed
187 188
  stage: manage
  group: import
189
  public: true
Jerome Ng's avatar
Jerome Ng committed
190
  pi_type: GMAU
191
  product_analytics_type: Both
192
  is_primary: true
193 194 195 196
  is_key: false
  health:
    level: 0
    reasons:
197
    - No target defined yet.
198
    - Target will be defined after the [metric instrumentation](https://gitlab.com/gitlab-org/gitlab/-/issues/297431) is implemented.
199
    - Improvement - Reducing friction for GitLab-to-GitLab migrations - The [MVC for the Group Migration feature](https://gitlab.com/groups/gitlab-org/-/epics/4374) which allows a much easier migration of an entire GitLab group will be delivered in `13.8`.
Jerome Ng's avatar
Jerome Ng committed
200
  implementation:
201
    status: Instrumentation
202
    reasons:
203
    - Instrumentation [originally completed](https://gitlab.com/gitlab-org/gitlab/-/issues/235424) in `13.5` looked incorrect (numbers very high) when reviewed in November. [Investigation](https://gitlab.com/gitlab-org/gitlab/-/issues/281604) of the data concluded that the original instrumentation was not implemented correctly and a [new instrumentation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297431) was created to fix the metric. 
204
    - Due to the current constraint in the bandwidth of the Manage:Import Backend Engineering team and recent increased volume of escalations from Support and the ProServ team, the new instrumentation issue is expected at the earliest in `13.8`. 
205
  metric_name: unique_users_all_imports (WIP - https://gitlab.com/gitlab-org/gitlab/-/issues/297431)
206
  
207 208 209 210
- name: Manage:Import - Other - Share of New Projects Imported
  base_path: "/handbook/product/performance-indicators/"
  definition: Share of projects created via import within 90 days of the namespace's
    creation.
211
  target: TBD - See Health section below for details
212
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
213
  section: dev
Jerome Ng's avatar
Jerome Ng committed
214 215
  stage: manage
  group: import
216
  public: true
Jerome Ng's avatar
Jerome Ng committed
217
  pi_type: Other
218
  product_analytics_type: Both
219 220 221 222 223
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
224
    - No target defined yet.
225
    - Target will be defined once the dashboard is [updated](https://gitlab.com/gitlab-data/analytics/-/issues/4244) with the improved metric (for both Saas and self-managed).
226
    - Improvement - Reducing friction for GitLab-to-GitLab migrations - The [MVC for the Group Migration feature](https://gitlab.com/groups/gitlab-org/-/epics/4374) which allows a much easier migration of an entire GitLab group will be delivered in `13.8`.
Jerome Ng's avatar
Jerome Ng committed
227
  implementation:
Haris Delalić's avatar
Haris Delalić committed
228
    status: Dashboard
229
    reasons:
230 231
    - Instrumented for [both GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/224758) in `13.6`.
    - Dashboard [update](https://gitlab.com/gitlab-data/analytics/-/issues/4244) is currently in progress.
232
  metric_name: TBD 
Jerome Ng's avatar
Jerome Ng committed
233

234
- name: Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
235 236
  base_path: "/handbook/product/performance-indicators/"
  definition: Number of unique users interacting with an issue (open, close, reopen,
237
    adjust label, adjust milestone, adjust weight, comment) 
238
  target: 484,510 for All, 148,176 for Paid 
239
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
240
  section: dev
Jerome Ng's avatar
Jerome Ng committed
241
  stage: plan
242
  group: project_management
243
  public: true
244
  pi_type: SMAU, Paid GMAU
245
  product_analytics_type: Both
246 247
  is_primary: true
  is_key: false
248 249
  lessons:
    learned:
250
      - Only 28% of instances are currently on a version with the new Plan xMAU metric counters. 
251
      - The ratio of [net new Paid GMAU to churned Paid GMAU](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)?widget=10634701&udv=0) improved from 3.75 in December to 3.99 in January, which indicates net growth and retention for Plan Paid users is healthy.
252
      - MAU interacting with Iterations increased by 52% from December to January. The next step to drive feature adoption is more holistically integrating Iterations into Boards. 
253 254 255
  monthly_focus:
    goals: 
      - Driver - Determine technical feasibility for allowing an [issue to be assigned to multiple groups](https://gitlab.com/gitlab-org/gitlab/-/issues/296668), which would likely solve [Plan adoption blockers (priorities 1 and 3)](https://gitlab.com/gitlab-org/gitlab/-/issues/218333).
256
      - Continue making progress on the [13.9 release plan](https://gitlab.com/gitlab-org/plan/-/issues/236) and roll into [13.10 goals](https://gitlab.com/gitlab-org/plan/-/issues/264).
257
      - Open > 3 documentation MRs in support of [better articulating Plan use cases](https://gitlab.com/gitlab-org/gitlab/-/issues/213810).
258
      - Complete the validation track for [saved filters/searches](https://gitlab.com/gitlab-org/gitlab/-/issues/14512), [customizable issue types](https://gitlab.com/gitlab-org/gitlab/-/issues/118666), and [interacting with a full issue from within a Board](https://gitlab.com/gitlab-org/gitlab/-/issues/13319)
259
      - Merge architectural blueprint for [consolidating Groups and Projects](https://gitlab.com/gitlab-org/architecture/tasks/-/issues/7) on the backend.
260 261 262
  health:
    level: 3
    reasons:
263
    - Achieved - FY21 Q4 targets of 437,490 for All, 101,835 for Paid. New targets set for FY22 Q1. 
264 265
    - Driver - For Paid, continue high level progress on [achieving feature parity between an Instance, Group, and Project](https://gitlab.com/groups/gitlab-org/-/epics/2885), [making issues more extensible](https://gitlab.com/groups/gitlab-org/-/epics/3354), and continuing to make incremental progress on [iterations](https://gitlab.com/groups/gitlab-org/-/epics/2422).
    - Improvement - For All, work towards addressing the [problem areas](https://gitlab.com/groups/gitlab-org/-/epics/4104) that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
266 267
    - Improvement - For All, continue making progress on [quality of life improvements](https://gitlab.com/groups/gitlab-org/-/epics/3216) for existing features. 
    - Improvement - For Paid, creating multi-level Epics spanning a Group hierarchy is clunky and non-intuitive. Improving this, along with making the Roadmap more interactive and intuitive would be a huge catalyst to drive Product Planning adoption. 
268
    - Improvement - For Paid, continue making incremental, iterative progress towards [Plan's core JTBD](/direction/plan/#problem-and-jobs-to-be-done) as outlined in the [Issue Tracking direction](/direction/plan/issue_tracking/). 
269 270
  implementation:
    status: Complete
271
    reasons:
272 273
    - xMAU data models are now updated with the new counter that aggregates unique users across many different issue actions.
    - New [Plan xMAU](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)) dashboard is live.
274
  metric_name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly 
275
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
276 277
    chart: 9967544
    dashboard: 758607
Jerome Ng's avatar
Jerome Ng committed
278 279
    embed: v2
  sisense_data_secondary:
Jerome Ng's avatar
Jerome Ng committed
280 281
    chart: 9980295
    dashboard: 758607
282
    embed: v2
283
  urls:
284
  - https://about.gitlab.com/direction/plan/issue_tracking/
285
  - https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)
Jerome Ng's avatar
Jerome Ng committed
286

Jerome Ng's avatar
Jerome Ng committed
287
- name: Plan:Product Planning - Paid GMAU - MAU creating Epics each Month
288 289
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users creating Epics each month.
290
  target: 3,900 Combined Unique Users for Self Managed and SaaS
291
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
292
  section: dev
Jerome Ng's avatar
Jerome Ng committed
293
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
294
  group: product_planning
295
  public: true
Jerome Ng's avatar
Jerome Ng committed
296
  pi_type: Paid GMAU
297
  product_analytics_type: Both
298 299 300 301 302
  is_primary: true
  is_key: false
  health:
    level: 3
    reasons:
Christen Dybenko's avatar
Christen Dybenko committed
303
    - Insight - GMAU is steadily rising with a known dip in August and Nov/Dec due to holidays. 
304
      [Monthly epic creation hit an all time high of +8500 in December](https://app.periscopedata.com/app/gitlab/602355/Portfolio-Management-G-SMAU-and-Feature-Success-Tracking?widget=9245316&udv=962108) and this was underscored 
Christen Dybenko's avatar
Christen Dybenko committed
305 306 307 308 309
      by a [jump in epic creation by Ultimate and Premium self managed customers](https://app.periscopedata.com/app/gitlab/602355/Product-Planning-G-SMAU-and-Feature-Success-Tracking?widget=7941113&udv=0) 
    - Improvement - [Epic Swimlanes](https://gitlab.com/groups/gitlab-org/-/epics/3352) shipped in 
      13.6 and we feel they be a significant driver of MAU growth for Product Planning nearterm (n2m) - giving 
      visibility to the feature on boards and adding additional value to better visualize your in-flight issues 
      through swimlanes.
310
    - Improvement - [Epic Boards](https://gitlab.com/groups/gitlab-org/-/epics/2864) is planned to ship in 13.9 and should continue the trend into the end of FY21 by enabling Portfolio-level planning.  
Jerome Ng's avatar
Jerome Ng committed
311 312
  implementation:
    status: Complete
313
    reasons:
314 315
    - Dashboarding - we've instrumented core metrics but are looking to expand the
      MAU definition to include all epic usage, not just creation, for a more accurate
316
      GMAU metric - [Issue](https://gitlab.com/gitlab-data/analytics/-/issues/5280). 
317
    - 2021-01 update - Similar to the Project Planning team, we have identified the [additional instrumentation](https://gitlab.com/gitlab-org/gitlab/-/issues/292253) needed to accomplish this a GMAU that includes all interactions with epics. 
318
  metric_name: epics 
319
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
320 321
    chart: 9980303
    dashboard: 758607
322 323
    embed: v2
  urls:
Christen Dybenko's avatar
Christen Dybenko committed
324
    
Mark Wood's avatar
Mark Wood committed
325
- name: Plan:Certify - Paid GMAU - MAU Interacting with Requirements
326 327 328 329 330
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users interacting (create, view, reference) each
    month
  target: TBD
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
331
  section: dev
Jerome Ng's avatar
Jerome Ng committed
332
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
333
  group: product_planning
334
  public: true
Jerome Ng's avatar
Jerome Ng committed
335
  pi_type: Paid GMAU
336
  product_analytics_type: Both
Jerome Ng's avatar
Jerome Ng committed
337
  is_primary: false
338 339
  is_key: false
  health:
Jerome Ng's avatar
Jerome Ng committed
340
    level: 0
341
    reasons:
Mark Wood's avatar
Mark Wood committed
342 343 344
    - Note - [Requirement interaction tracking](https://gitlab.com/gitlab-org/product-intelligence/-/issues/370) issue is complete. 
      Data is in the process of being aggregated. Early data shows healthy adoption of test report linking.
      [Tracking issue](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
345
    - Improvement - UX improvements for [Requirements Management](https://gitlab.com/groups/gitlab-org/-/epics/3708)
Mark Wood's avatar
Mark Wood committed
346
      based on completed research study, Requirement [Export MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/290813).
Jerome Ng's avatar
Jerome Ng committed
347 348
  implementation:
    status: Dashboard
349
    reasons:
Justin Farris's avatar
Justin Farris committed
350
    - Dashboarding - Requirement Creation is instrumented is complete. Tracking issue for gmau dashboard [here](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
351
  metric_name: TBD
Mark Wood's avatar
Mark Wood committed
352
- name: Plan:Certify - Other - Number of Requirements Created Monthly
353 354
  base_path: "/handbook/product/performance-indicators/"
  definition: A monthly count of the number of requirements created.
Mark Wood's avatar
Mark Wood committed
355
  target: 1000 by end of March 2021
356
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
357
  section: dev
Jerome Ng's avatar
Jerome Ng committed
358
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
359
  group: product_planning
360
  public: true
Jerome Ng's avatar
Jerome Ng committed
361
  pi_type: Other
362
  product_analytics_type: Both
363 364 365 366 367
  is_primary: false
  is_key: false
  health:
    level: 3
    reasons:
Mark Wood's avatar
Mark Wood committed
368 369
    - Improvement - [Requirement Export MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/290813) feature to release in 13.8
    - Insight - Requirement creation has decreased. Investigation shows a substantial set of users are Universities, which explains the downturn for the holidays. 
Mark Wood's avatar
Mark Wood committed
370
    - Insight - Self-managed instances creating requirements continues to grow.   
Jerome Ng's avatar
Jerome Ng committed
371 372
  implementation:
    status: Dashboard
373
    reasons:
374
    - Dashboarding -  accurate instrumentation on Self Managed, still needs verification
375
      for SaaS - [issue](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
Jerome Ng's avatar
Jerome Ng committed
376
  metric_name: requirements_created
377
  sisense_data:
Mark Wood's avatar
Mark Wood committed
378
    chart: 9666808
379
    dashboard: 634539
380 381 382
    embed: v2
  urls:
  - https://gitlab.com/gitlab-data/analytics/-/issues/5270
383

James Ramsay's avatar
James Ramsay committed
384
- name: Create - SMAU - Unique users writing to a Git repository
385
  base_path: "/handbook/product/performance-indicators/"
386 387
  definition: A rolling count of the number of unique users who performed any Git write operation within the last 28 days.
  target: |
James Ramsay's avatar
James Ramsay committed
388
    Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
389
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
390
  section: dev
Jerome Ng's avatar
Jerome Ng committed
391
  stage: Create
392
  public: true
Jerome Ng's avatar
Jerome Ng committed
393
  pi_type: SMAU
394
  product_analytics_type: Both
395 396 397
  is_primary: true
  is_key: false
  health:
398
    level: 3
399
    reasons:
James Ramsay's avatar
James Ramsay committed
400 401 402 403 404
    - Observation - December 2020 SMAU is less than November 2020 SMAU. A
      similar drop can be observed in GMAU as well, suggesting a common cause.
      The cause is likely seasonality consistent with the end of year. For this
      reason January SMAU and GMAU is expected to return to November 2020
      levels and continue growing.
James Ramsay's avatar
James Ramsay committed
405 406 407 408 409 410
    - Insight - on the median GitLab instance, 33% of users perform a Git write
      operation each month
      ([source](https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=10435358&udv=1100338)).
    - Insight - on the median Paid GitLab instance (self-managed), 50% of users
      perform a Git write operation each month
      ([source](https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=10435361&udv=1100338)).
Jerome Ng's avatar
Jerome Ng committed
411 412
  implementation:
    status: Complete
413
    reasons:
James Ramsay's avatar
James Ramsay committed
414 415 416 417
    - Improvement - AMAU is currently over counted. A deduplicated counter was implemented
      [gitlab#234027](https://gitlab.com/gitlab-org/gitlab/-/issues/234027) in GitLab 13.6.
      The charts do not yet reflect the new counter
      [gitlab#291943](https://gitlab.com/gitlab-org/gitlab/-/issues/291943).
418 419
  metric_name: action_monthly_active_users_project_repo
  sisense_data:
Kathleen Tam's avatar
Kathleen Tam committed
420 421
    chart: 9967642
    dashboard: 758607
422
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
423 424 425 426
  sisense_data_secondary:
    chart: 9980442
    dashboard: 758607
    embed: v2
427
  urls:
Daniel Gruesso's avatar
Daniel Gruesso committed
428
  - https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard
429

James Ramsay's avatar
James Ramsay committed
430
- name: Create:Source Code - GMAU - Unique users writing to a project Git repository
431
  base_path: "/handbook/product/performance-indicators/"
432 433 434
  definition: A rolling count of the number of unique users who performed any Git write
    operation within the last 28 days
  target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
435
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
436
  section: dev
Jerome Ng's avatar
Jerome Ng committed
437 438
  stage: create
  group: source_code
439
  public: true
Jerome Ng's avatar
Jerome Ng committed
440
  pi_type: GMAU
441
  product_analytics_type: Both
442 443 444
  is_primary: true
  is_key: false
  health:
Daniel Gruesso's avatar
Daniel Gruesso committed
445
    level: 3
446
    reasons:
447
    - Starting in [GitLab 13.6](https://gitlab.com/gitlab-org/gitlab/-/issues/234027)
448
      AMAU for this metric takes into consideration all repository types.
449 450 451 452
    - Improvement - work towards providing users more insights while navigating their
      code with [Code Intelligence](https://gitlab.com/groups/gitlab-org/-/epics/1576)
    - Improvement - work towards providing more control over [managing repository
      size](https://gitlab.com/groups/gitlab-org/-/epics/3793)
Jerome Ng's avatar
Jerome Ng committed
453
  implementation:
454
    status: Complete
455
    reasons:
456
    - Enabled on GitLab.com
457 458
    - Released for self-managed in GitLab 13.3
  metric_name: action_monthly_active_users_project_repo
Daniel Gruesso's avatar
Daniel Gruesso committed
459
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
460 461 462 463 464 465
    chart: 9967642
    dashboard: 758607
    embed: v2
  sisense_data_secondary:
    chart: 9980442
    dashboard: 758607
Daniel Gruesso's avatar
Daniel Gruesso committed
466
    embed: v2
467
  urls:
Daniel Gruesso's avatar
Daniel Gruesso committed
468
  - https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=9520571&udv=0
James Ramsay's avatar
James Ramsay committed
469
  - https://gitlab.com/gitlab-org/gitlab/-/issues/234027
470

471
- name: Create:Source Code Secondary - GMAU - Unique users approving merge requests
472
  base_path: "/handbook/product/performance-indicators/"
473
  definition: A rolling count of the number of unique users who approved any merge request within the last 28 days
474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: source_code
  public: true
  pi_type: GMAU
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Not currently instrumented
    - Currently planning instrumentation and evaluating if this is a more relevant metric for the Source Code group
  urls:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/293014

491
- name: Create:Source Code Tertiary - GMAU - Unique users pushes rejected by push rules
492
  base_path: "/handbook/product/performance-indicators/"
493
  definition: A rolling count of the number of unique users who had one or more push rejected by any push rule within the last 28 days
494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: source_code
  public: true
  pi_type: GMAU
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Not currently instrumented
    - Currently planning instrumentation and evaluating if this is a more relevant metric for the Source Code group
  urls:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/293016

James Ramsay's avatar
James Ramsay committed
511
- name: Create:Code Review - GMAU - Unique users using Merge Requests
512
  base_path: "/handbook/product/performance-indicators/"
513 514
  definition: This is defined as users who perform one or more of the following actions in
   a merge request - create, merge, or close a Merge Request.
Kai Armstrong's avatar
Kai Armstrong committed
515
  target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU. 
516
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
517
  section: dev
Jerome Ng's avatar
Jerome Ng committed
518
  stage: create
519
  group: code_review
520
  public: true
James Ramsay's avatar
James Ramsay committed
521
  pi_type: GMAU
522
  product_analytics_type: Both
523
  is_primary: true
524 525
  is_key: false
  health:
526
    level: 3
527
    reasons:
Kai Armstrong's avatar
Kai Armstrong committed
528
    - Target - This aligns with our current understanding that Code Review is about 50% of the MAU of the stage metric and continuing to maintain that difference.
529
    - Insight - Paid users of Code Review make up about one-third of total Code Review users - this provides an opportunity to convert to paid and increase users.
530
    - Improvement - Support for [custom commit messages when applying a suggested change](https://gitlab.com/gitlab-org/gitlab/-/issues/25381) will allow more teams with strict commit message guidelines to adopt this feature and maintain conformity with the project.
531 532
    - Improvement - [Mark changes as viewed](https://gitlab.com/gitlab-org/gitlab/-/issues/17531/) in merge requests is a foundational step towards tracking and communicating progress on merge requests to other participants.
    - Improvement - SCM <-> Code Review <-> Verify correlation charts now defined in [new charts](#createcode-review---other---code-reviewscm-ratio-by-umau-and-subscription-tier)
Jerome Ng's avatar
Jerome Ng committed
533
  implementation:
534
    status: Complete
535
    reasons:
536 537
    - Enabled on GitLab.com 2020-07-04
    - Enabled for self-managed on GitLab 13.2
538
    - Working on more comprehensive MAU tracking in [&4948](https://gitlab.com/groups/gitlab-org/-/epics/4948)
539
  metric_name: merge_requests_users 
Daniel Gruesso's avatar
Daniel Gruesso committed
540
  sisense_data:
541 542
    chart: 10426645
    dashboard: 786738
Daniel Gruesso's avatar
Daniel Gruesso committed
543
    embed: v2
544
  sisense_data_secondary:
545 546
    chart: 10624715
    dashboard: 786738
547
    embed: v2
548
  urls:
549
  - https://app.periscopedata.com/app/gitlab/526749/Create---Code-Review-category-dashboard
James Ramsay's avatar
James Ramsay committed
550

551
- name: Create:Code Review - Other - Code Review/SCM Ratio by UMAU and Subscription Tier
Kai Armstrong's avatar
Kai Armstrong committed
552 553
  base_page: "/handbook/product/performance-indicators/"
  definition: 
554
  target: TBD - Primarily this metric is used for monitoring the relationship between SCM GMAU and Code Review GMAU to make sure it doesn't not fall out of alignment.
Kai Armstrong's avatar
Kai Armstrong committed
555 556 557 558 559 560 561 562 563 564 565 566 567
  org: Dev Section
  section: dev
  stage: create
  group: code_review
  public: true
  pi_type: Other
  product_analytics_type: Both
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Target - this metric is primarily monitored to remain at a consistent level.
568 569
    - Insight - Paid Instances with 10+ users more closely track to 75% of SCM Users using Code Review
    - Insight - Free Instances and those with less than 10 users drive down the SCM<->Code Review Ratio as most have less than 50% adoption
Kai Armstrong's avatar
Kai Armstrong committed
570 571 572 573
  implementation:
    status: Complete
  metric_name: merge_requests_users
  sisense_data:
574 575 576 577 578 579
    chart: 10728688
    dashboard: 801102
    embed: v2
  sisense_data_secondary:
    chart: 10688270
    dashboard: 801102
Kai Armstrong's avatar
Kai Armstrong committed
580
    embed: v2
581 582
  urls:
  - https://app.periscopedata.com/app/gitlab/801102/SCM-%3C%3E-Code-Review-%3C%3E-Verify
583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607
  - https://app.periscopedata.com/app/gitlab/801102/SCM-%3C%3E-Code-Review-%3C%3E-Verify?widget=10728688&udv=1218043

- name: Create:Code Review - Editor Extension (VS Code) - Category MAU
  base_path: "/handbook/product/performance-indicators/"
  definition: This is defined as users who use [GitLab Workflow](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow) for VS Code and make at least one API request to a GitLab instance.
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: code_review
  public: true
  pi_type: Other
  product_analytics_type: Both
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Instrumentation in 13.9
    - Improvement - [GitLab Workflow 3.11.0](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CHANGELOG.md#3110-2021-01-25) released with auto completion for GitLab CI Variables
    - Improvement - [GitLab Workflow 3.10.0](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/CHANGELOG.md#3100-2021-01-19) released with viewing code review comments on merge request diffs
  implementation:
    status: Instrumentation
    reasons:
    - MAU Collection [Issue](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/issues/258)
Kai Armstrong's avatar
Kai Armstrong committed
608

609
- name: Create:Gitaly - Other - Availability
610
  base_path: "/handbook/product/performance-indicators/"
611 612 613 614 615 616 617
  definition: Percentage of time during which Gitaly is fully operational and providing
    service to users within SLO parameters. Availability is measured according by
    Apdex (latency) and error rate. Gitaly is considered available when at least 50%
    of users are experiencing satisfactory latency, _and_ 50% of requests are completing
    successfully. If either of these conditions is not met, the service is experiencing
    an outage.
  target: 99.95%, consistent with GitLab.com availability service level objectives
618
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
619
  section: dev
620 621
  stage: create
  group: gitaly
622
  public: true
Jeremy Watson (ex-GitLab)'s avatar
Jeremy Watson (ex-GitLab) committed
623
  pi_type: Other
624
  product_analytics_type: Both
625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652
  is_primary: true
  is_key: false
  health:
    level: 0
    reasons:
    - Instrumentation for this metric started in 13.4 and is still ongoing.
  implementation:
    status: Instrumentation
    reasons:
    - Rolling 7 day metrics available in [Grafana](https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview)
      for GitLab.com. Current availability per Adpex is 99.98%.
    - Improvement - self-managed availability metrics are [planned](https://gitlab.com/gitlab-org/gitlab/-/issues/216683)
  metric_name: gitaly_apdex
  urls:
  - https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
  - https://gitlab.com/gitlab-org/gitlab/-/issues/216683

- name: Create:Gitaly - Other - Unique instances with Gitaly Cluster enabled
  base_path: "/handbook/product/performance-indicators/"
  definition: Number of recorded GitLab instances with Gitaly Cluster enabled
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: gitaly
  public: true
  pi_type: Other
  product_analytics_type: Self-Managed
Christen Dybenko's avatar
Christen Dybenko committed
653
  is_primary: false
654 655
  is_key: false
  health:
656
    level: 2
657
    reasons:
658 659
    - No target defined as the feature is still being rolled out.
    - Improvement - introduce [Variable Replication Factor and auto-reblancing within a cluster](https://gitlab.com/groups/gitlab-org/-/epics/3372) to enable more enterprise-level adoption.
Jerome Ng's avatar
Jerome Ng committed
660
  implementation:
661
    status: Complete
662
    reasons:
Mark Wood's avatar
Mark Wood committed
663
    - Insight - Reference architecture has been updated with recommendations for Gitaly Cluster for 13.8.
664 665
    - Instrumentation for reporting on instances with Gitaly Cluster enabled was introduced in 13.0 with the GA of Gitaly Cluster.
  metric_name: gitaly_clusters
Jerome Ng's avatar
Jerome Ng committed
666
  sisense_data:
667 668
    chart: 9504807
    dashboard: 728407
Jerome Ng's avatar
Jerome Ng committed
669
    embed: v2
670
  urls:
671
  - https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
James Ramsay's avatar
James Ramsay committed
672 673

- name: Create:Editor - GMAU - Unique users committing via any web editor
674 675
  base_path: "/handbook/product/performance-indicators/"
  definition: A rolling count of the number of unique users who performed a Git write
676 677
    operation to any respository using the Web IDE, Web Editor or Snippet within a
    stage in the last 28 days
Eric Schurter's avatar
Eric Schurter committed
678
  target: 4% growth month over month for All 
679
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
680
  section: dev
Jerome Ng's avatar
Jerome Ng committed
681 682
  stage: create
  group: editor
683
  public: true
Jerome Ng's avatar
Jerome Ng committed
684
  pi_type: GMAU
685
  product_analytics_type: Both
686
  is_primary: true
687 688
  is_key: false
  health:
689
    level: 2
690
    reasons:
691
    - New target GMAU to be set once we get cumulative, de-duped data from newly aligned categories.
692 693
    - Insight - Rebound from the holiday season to nearly November levels. We'll be monitoring to ensure consistent organic growth in subsequent months. 
    - Improvement - No targeted efforts in the next few releases to specifically grow GMAU, but currently working to [migrate the Web IDE to use Editor Lite](https://gitlab.com/gitlab-org/gitlab/-/issues/297449) which will bring consistency across editors and a [moderate performance improvement](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51212#note_483338464). 
Jerome Ng's avatar
Jerome Ng committed
694
  implementation:
Jerome Ng's avatar
Jerome Ng committed
695
    status: Complete
696
    reasons:
697
    - Instrumentation of [Static Site Editor and Wiki to roll up into GMAU](https://gitlab.com/gitlab-org/gitlab/-/issues/283947) completed in 13.7, waiting for data to be reflected in the charts.
698
  metric_name: action_monthly_active_users_ide_edit 
699
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
700 701
    chart: 9967737
    dashboard: 758607
702
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
703 704 705
  sisense_data_secondary:
    chart: 9980340
    dashboard: 758607