dev_section.yml 45.6 KB
Newer Older
Eric Brinkman's avatar
Eric Brinkman committed
1
- name: Dev - Section TMAU - Estimated & Recorded Sum of Dev Section SMAU
2 3
  base_path: "/handbook/product/performance-indicators/"
  definition: A sum of all SMAUs across the three stages in the Dev Section (Manage,
Jerome Ng's avatar
Jerome Ng committed
4
    Plan, and Create) for All users and Paid users.
Eric Brinkman's avatar
Eric Brinkman committed
5
  target: Estimated TMAU growth 4% MoM; 2.73M by end of FY21Q4 for TMAU, 603K by end of FY21Q4 for Paid TMAU
6
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
7
  section: dev
8
  public: true
Jerome Ng's avatar
Jerome Ng committed
9
  pi_type: Section TMAU
10
  product_analytics_type: Both
11 12 13
  is_primary: true
  is_key: false
  health:
14
    level: 3
15
    reasons:
Eric Brinkman's avatar
Eric Brinkman committed
16 17 18
    - We saw a 3.5% drop in Dev Section TMAU between Nov 2020 and Dec 2020. UMAU dropped 5.2% as a reference point.
    - Despite the drop, the Dev section has already achieved our end of Q4 target of 2.75M TMAU by 300K users.
    - TMAU drops were the largest for Create. Plan declined slightly by 3K users and Manage increased by 8K users, most likely because of upgrades as the GMAU was only turned on by default in 13.5.
19
    - Self-managed TMAU increased, but SaaS significantly decreased, likely to free users declining due to seasonality - as a next step, we'll work to add a 4th graph in this section with TMAU broken out by tiers, including free and paid.
Eric Brinkman's avatar
Eric Brinkman committed
20 21
    - Insight - instances with large amounts of users are unlikely to upgrade immediately, typically taking a few months to do so. So, our larger instances were likely counted as estimated earlier and as the larger instances upgrade, what we'll likely see is that estimated uplift was under reported early. It's difficult to tell if it will be over reported as our largest self-managed instances are unlikely to ever report back. It will be interesting to see how the uplifts change when we base the uplift on users instead of subscriptions.
    - Improvement - For All Usage, address the [problems](https://gitlab.com/groups/gitlab-org/-/epics/4104)
22
      that emerged out of the Category Maturity Scorecard exercise for Issue Tracking
Eric Brinkman's avatar
Eric Brinkman committed
23
      and continue making progress on [usability improvements](https://gitlab.com/groups/gitlab-org/-/epics/3216)
24
      for existing Plan features.
Eric Brinkman's avatar
Eric Brinkman committed
25 26 27 28 29 30
    - Improvement - For All Usage, Merge Request [usability improvements](https://gitlab.com/groups/gitlab-org/-/epics/3607)
    - Improvement - For All Usage, [MR reviewers](https://gitlab.com/gitlab-org/gitlab/-/issues/237921) will make us more competitive with GitHub and should help with the lovability of assigning merge requests
    - Improvement - For All Usage, [Iterations in GitLab](https://gitlab.com/groups/gitlab-org/-/epics/2422) will make plan more usable for many Agile development teams
    - Improvement - For Paid Usage, [Epic Boards](https://gitlab.com/groups/gitlab-org/-/epics/4117) will allow for much better portfolio management via boards
    - Improvement - For Paid Usage, [SSO Group Sync](https://gitlab.com/gitlab-org/gitlab/-/issues/118)
    - Improvement - For All Usage, [Workspaces](https://gitlab.com/groups/gitlab-org/-/epics/4257)
Eric Brinkman's avatar
Eric Brinkman committed
31 32
    - Improvement - For All Usage, [Groups/Project Simplification](https://gitlab.com/groups/gitlab-org/-/epics/2885)
    - Improvement - For All Usage, [Settings and Navigation improvements](https://gitlab.com/groups/gitlab-org/-/epics/4831)
Jerome Ng's avatar
Jerome Ng committed
33
  implementation:
34
    status: Complete
Eric Brinkman's avatar
Eric Brinkman committed
35
    reasons:
36
  metric_name: issues, action_monthly_active_users_project_repo, analytics_unique_visits_for_any_target
Eric Brinkman's avatar
Eric Brinkman committed
37
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
38 39
    chart: 9982128
    dashboard: 758607
40 41
    embed: v2
  sisense_data_secondary:
42 43
    chart: 10138877
    dashboard: 758607
Eric Brinkman's avatar
Eric Brinkman committed
44
    embed: v2
Mathieu Peychet's avatar
Mathieu Peychet committed
45 46 47 48
  sisense_data_tertiary:
    chart: 9992267
    dashboard: 758607
    embed: v2
49

Jeremy Watson (ex-GitLab)'s avatar
Jeremy Watson (ex-GitLab) committed
50
- name: Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features
51
  base_path: "/handbook/product/performance-indicators/"
52
  definition: The number of unique users viewing any <a href="https://docs.gitlab.com/ee/user/analytics/">Analytics page</a>.
53
  target: 116,050
54
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
55
  section: dev
Jerome Ng's avatar
Jerome Ng committed
56
  stage: manage
57
  group: analytics
58
  public: true
59
  pi_type: SMAU, GMAU
60
  product_analytics_type: Both
61
  is_primary: true
62 63
  is_key: false
  health:
64
    level: 3
65
    reasons:
66 67 68 69 70 71
    - Insights - Optimize feature usage is now above target. 
    - Insights - SaaS usage of Optimize features dropped 10% in December. The most significant drop was a 13.7% decrease in views of CI/CD analytics by SaaS users. In December and previous months this page would often give a 500 error. This looks like it has been resolved now.
    - Insights - Self-managed usage of Optimize features surpassed SaaS usage for the first time in December. This was due to a combination of a decline in usage by SaaS users and an increase in usage by self-managed users.
    - Insights - Usage by self-managed users increased 23% in December. We have been focused on improving instance-level analytics over the past 3 milestones and the data shows that these features are being adopted. In fact, most of the increase in self-managed usage is the result of interest in instance-level analytics. DevOps Reports usage increased by 35% in December. Instance-level Usage Trends was announced in 13.6 and is gaining popularity. DevOps Score and Usage Trends combined accounted for 50% of the increase in usage of Optimize features on self managed. Instance-level analytics are still only available to Admin users on self-managed instances. As we make these available to SaaS users in 2021 and continue to iterate on them, we expect to see further adoption and an overall improvement in the Optimize PI.
    - Improvements - DevOps Adoption encountered a setback that has delayed its rollout. There has been interest in this feature from enterprise customers wanting insights into how GitLab is being used in their organizations. We are investigating the best approach to moving forward on this feature.
    - Improvements - Access to the new instance-level features are still restricted to Administrators on self-managed instances. We are evaluating options to make these more widely available and expect to make progress on this in 13.9. 
Jerome Ng's avatar
Jerome Ng committed
72
  implementation:
73
    status: Complete
74
  metric_name: analytics_total_unique_counts_monthly
75
  sisense_data:
Kathleen Tam's avatar
Kathleen Tam committed
76 77
    chart: 10039390
    dashboard: 758607
78
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
79 80 81 82 83
  sisense_data_secondary:
    chart: 10039413
    dashboard: 758607
    embed: v2

84 85 86
- name: Manage:Access - Paid GMAU - MAU Using Paid SAML
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users logging into GitLab using SAML SSO for Groups.
87
  target: 30000. See https://gitlab.com/gitlab-com/Product/-/issues/1832 for further detail.
88
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
89
  section: dev
Jerome Ng's avatar
Jerome Ng committed
90 91
  stage: manage
  group: access
92
  public: true
Jerome Ng's avatar
Jerome Ng committed
93
  pi_type: Paid GMAU
Melissa Ushakov's avatar
Melissa Ushakov committed
94
  product_analytics_type: SaaS
95 96 97
  is_primary: true
  is_key: false
  health:
98
    level: 3
99
    reasons:
Melissa Ushakov's avatar
Melissa Ushakov committed
100 101 102
    - Improvements - In 13.8, we will allow group owners to [bypass SSO enforcement](https://gitlab.com/gitlab-org/gitlab/-/issues/211962) to prevent accidental lockout during the SAML setup process.
    - Improvements - In 13.8, we will automatically triggering a request to the users SAML SSO IdP when their session expires instead of requiring a user to click to trigger a check. We can then allow groups to [customize their SSO timeouts]() and check for a valid session more often.
    - Insights - We exceeded our 30k group SAML logins per month target! You can find more details this [issue](https://gitlab.com/gitlab-com/Product/-/issues/1949). I've set that to be the target for this quarter. 
Melissa Ushakov's avatar
Melissa Ushakov committed
103
    - Insights - I have a list of customers who match the profile of SSO adopter who are not using SSO yet. I've begun requesting TAMs feedback in this [issue](https://gitlab.com/gitlab-com/Product/-/issues/1949)
Jerome Ng's avatar
Jerome Ng committed
104
  implementation:
Kathleen Tam's avatar
Kathleen Tam committed
105
    status: Complete 
106
    reasons:
Melissa Ushakov's avatar
Melissa Ushakov committed
107
  metric_name: user_auth_by_provider (Group_SAML)
Kathleen Tam's avatar
Kathleen Tam committed
108 109 110 111
  sisense_data:
    chart: 10039419
    dashboard: 758607
    embed: v2
112 113 114 115
- name: Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users viewing Audit Events, Compliance Dashboard,
    or Credential Inventory.
116
  target: TBD
117
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
118
  section: dev
Jerome Ng's avatar
Jerome Ng committed
119 120
  stage: manage
  group: compliance
121
  public: true
Jerome Ng's avatar
Jerome Ng committed
122
  pi_type: Paid GMAU
123
  product_analytics_type: Both
124 125 126
  is_primary: true
  is_key: false
  health:
127
    level: 1
128
    reasons:
129 130
    - We're still [building out the compliance features](https://gitlab.com/gitlab-org/manage/compliance/general/-/issues/13) that are tracked by this metric. This improvement work will take longer than desired to complete since we do not have enough people to ship any faster. We expect iterations to ship on a continuous basis across each of these features.
    - Setting a target is premature since we're still collecting initial data. We would like to collect a total of 6 months worth of data before setting a target. 
131
    - Result: Increased GMAU by ~25% in December [by making Audit Events available to Developers and making the feature more discoverable](https://gitlab.com/gitlab-org/gitlab/-/issues/257514).
132 133 134 135
    - Adding a [Change Management view](https://gitlab.com/groups/gitlab-org/-/epics/4802) to the Compliance Dashboard. Planned for `Q2 2021`.
    - Improving PAT & SSH [credential management](https://gitlab.com/groups/gitlab-org/-/epics/3084). Ongoing with expected epic completion in `Q2 2021`.
    - Adding [additional credentials](https://gitlab.com/groups/gitlab-org/-/epics/4110) to the credential inventory. Planned for `Q3 2021`
    - Improving [discoverability](https://gitlab.com/groups/gitlab-org/-/epics/4441) of Compliance features. Planned for `Q2 2021`.
Jerome Ng's avatar
Jerome Ng committed
136
  implementation:
137
    status: Complete 
138
    reasons:
139
  metric_name: i_compliance_audit_events, g_compliance_audit_events, g_compliance_dashboard, i_compliance_credential_inventory, usage_data_g_compliance_dashboard, usage_data_a_compliance_audit_events_api
140
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
141 142
    chart: 9980315
    dashboard: 758607
143
    embed: v2
144 145 146 147
  sisense_data_secondary:
    chart: 10039422
    dashboard: 758607
    embed: v2
148
    
149
- name: Manage:Compliance - Other - Projects Marked with a Compliance Framework
150
  base_path: "/handbook/product/performance-indicators/"
151
  definition: New and cumulative number of projects using compliance frameworks.
152
  target: 1,030 total new projects in Q4 using compliance frameworks (SaaS and self-managed combined)
153
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
154
  section: dev
Jerome Ng's avatar
Jerome Ng committed
155 156
  stage: manage
  group: compliance
157
  public: true
Jerome Ng's avatar
Jerome Ng committed
158
  pi_type: Other
159
  product_analytics_type: Both
160
  is_primary: false
161 162
  is_key: false
  health:
163
    level: 2
164
    reasons:
165
    - We're [investigating](https://gitlab.com/gitlab-data/analytics/-/issues/7302) the sudden data disruption for this chart.
166 167
    - In Q3 we saw 936 new projects use compliance frameworks. These were a limited selection of 5, hard-coded options. We are targeting a 10% increase in adoption by making these frameworks [customizable](https://gitlab.com/gitlab-org/gitlab/-/issues/255340)
    - Additionally, we're using a conservative target for now while we focus on [several improvements](https://gitlab.com/groups/gitlab-org/-/epics/4082) to the underlying feature that drives this metric.
168 169 170 171
    - We're continuing iteration on [compliance frameworks](https://gitlab.com/groups/gitlab-org/-/epics/4082) to give them more utility and solve several standardization and enforcement problems for customers. 
    - We're adding an improvement to give organizations [better control](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) of who can add or modify these frameworks at the project level in `13.8`.
    - We expect the next iteration, consisting of [frontend](https://gitlab.com/gitlab-org/gitlab/-/issues/292696) and [backend](https://gitlab.com/gitlab-org/gitlab/-/issues/254389) additions to ship in `13.9`. This iteration is particularly important for the reason below.
    - By `13.10` we expect compliance frameworks to fully support one of our most requested solutions, [group compliance pipelines](https://gitlab.com/groups/gitlab-org/-/epics/3156), as a required step to implement it, which we expect to drive even more adoption of compliance frameworks.
Jerome Ng's avatar
Jerome Ng committed
172
  implementation:
173
    status: Complete
174
    reasons:
175
  metric_name: projects_with_compliance_framework
176
  sisense_data:
Matt Gonzales (ex-GitLab)'s avatar
Matt Gonzales (ex-GitLab) committed
177
    chart: 9071082
178 179
    dashboard: 663045
    embed: v2
180

181 182 183
- name: Manage:Import - GMAU - MAU importing
  base_path: "/handbook/product/performance-indicators/"
  definition: A monthly count of the unique number of users performing an import.
184
  target: TBD - See Health section below for details
185
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
186
  section: dev
Jerome Ng's avatar
Jerome Ng committed
187 188
  stage: manage
  group: import
189
  public: true
Jerome Ng's avatar
Jerome Ng committed
190
  pi_type: GMAU
191
  product_analytics_type: Both
192
  is_primary: true
193 194 195 196
  is_key: false
  health:
    level: 0
    reasons:
197
    - No target defined yet.
198
    - Target will be defined after the [metric instrumentation](https://gitlab.com/gitlab-org/gitlab/-/issues/297431) is implemented.
199
    - Improvement - Reducing friction for GitLab-to-GitLab migrations - The [MVC for the Group Migration feature](https://gitlab.com/groups/gitlab-org/-/epics/4374) which allows a much easier migration of an entire GitLab group will be delivered in `13.8`.
Jerome Ng's avatar
Jerome Ng committed
200
  implementation:
201
    status: Instrumentation
202
    reasons:
203
    - Instrumentation [originally completed](https://gitlab.com/gitlab-org/gitlab/-/issues/235424) in `13.5` looked incorrect (numbers very high) when reviewed in November. [Investigation](https://gitlab.com/gitlab-org/gitlab/-/issues/281604) of the data concluded that the original instrumentation was not implemented correctly and a [new instrumentation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297431) was created to fix the metric. 
204
    - Due to the current constraint in the bandwidth of the Manage:Import Backend Engineering team and recent increased volume of escalations from Support and the ProServ team, the new instrumentation issue is expected at the earliest in `13.8`. 
205
  metric_name: unique_users_all_imports (WIP - https://gitlab.com/gitlab-org/gitlab/-/issues/297431)
206
  
207 208 209 210
- name: Manage:Import - Other - Share of New Projects Imported
  base_path: "/handbook/product/performance-indicators/"
  definition: Share of projects created via import within 90 days of the namespace's
    creation.
211
  target: TBD - See Health section below for details
212
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
213
  section: dev
Jerome Ng's avatar
Jerome Ng committed
214 215
  stage: manage
  group: import
216
  public: true
Jerome Ng's avatar
Jerome Ng committed
217
  pi_type: Other
218
  product_analytics_type: Both
219 220 221 222 223
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
224
    - No target defined yet.
225
    - Target will be defined once the dashboard is [updated](https://gitlab.com/gitlab-data/analytics/-/issues/4244) with the improved metric (for both Saas and self-managed).
226
    - Improvement - Reducing friction for GitLab-to-GitLab migrations - The [MVC for the Group Migration feature](https://gitlab.com/groups/gitlab-org/-/epics/4374) which allows a much easier migration of an entire GitLab group will be delivered in `13.8`.
Jerome Ng's avatar
Jerome Ng committed
227
  implementation:
Haris Delalić's avatar
Haris Delalić committed
228
    status: Dashboard
229
    reasons:
230 231
    - Instrumented for [both GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/224758) in `13.6`.
    - Dashboard [update](https://gitlab.com/gitlab-data/analytics/-/issues/4244) is currently in progress.
232
  metric_name: TBD 
Jerome Ng's avatar
Jerome Ng committed
233

234
- name: Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
235 236
  base_path: "/handbook/product/performance-indicators/"
  definition: Number of unique users interacting with an issue (open, close, reopen,
237
    adjust label, adjust milestone, adjust weight, comment) 
238
  target: 437,490 for All, 101,835 for Paid 
239
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
240
  section: dev
Jerome Ng's avatar
Jerome Ng committed
241
  stage: plan
242
  group: project_management
243
  public: true
244
  pi_type: SMAU, Paid GMAU
245
  product_analytics_type: Both
246 247
  is_primary: true
  is_key: false
248 249 250 251 252 253 254 255 256 257 258
  lessons:
    learned:
      - The decline in SaaS for December does not seem isolated to Plan as nearly every group has a lower SaaS xMAU for the month. This is likely attributable more people taking extended vacations during the holiday season.
      - The top 50 Plan paid self-managed customers, as well as the paid customers with the most expansion and shrinkage MoM. See the bottom of [this dashboard](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly))
      - The "health status" feature usage is [almost non-existent](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)?widget=10444593&udv=0) and we should consider down-tiering or continuing to iterate on it by automating it and integrating it into Boards and Roadmaps. 
      - Quantitative issue field level data validates that less frequently used fields -- such as setting confidentiality and locking an issue -- should be [moved into the secondary menu](https://gitlab.com/gitlab-org/gitlab/-/issues/296959) within the issue header to decrease clutter in the sidebar.
  monthly_focus:
    goals: 
      - Driver - Determine technical feasibility for allowing an [issue to be assigned to multiple groups](https://gitlab.com/gitlab-org/gitlab/-/issues/296668), which would likely solve [Plan adoption blockers (priorities 1 and 3)](https://gitlab.com/gitlab-org/gitlab/-/issues/218333).
      - Start work on the [13.9 release plan](https://gitlab.com/gitlab-org/plan/-/issues/236).
      - Open > 3 documentation MRs in support of [better articulating Plan use cases](https://gitlab.com/gitlab-org/gitlab/-/issues/213810).
259 260 261
  health:
    level: 3
    reasons:
262 263
    - Driver - For Paid, continue high level progress on [achieving feature parity between an Instance, Group, and Project](https://gitlab.com/groups/gitlab-org/-/epics/2885), [making issues more extensible](https://gitlab.com/groups/gitlab-org/-/epics/3354), and continuing to make incremental progress on [iterations](https://gitlab.com/groups/gitlab-org/-/epics/2422).
    - Improvement - For All, work towards addressing the [problem areas](https://gitlab.com/groups/gitlab-org/-/epics/4104) that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
264 265
    - Improvement - For All, continue making progress on [quality of life improvements](https://gitlab.com/groups/gitlab-org/-/epics/3216) for existing features. 
    - Improvement - For Paid, creating multi-level Epics spanning a Group hierarchy is clunky and non-intuitive. Improving this, along with making the Roadmap more interactive and intuitive would be a huge catalyst to drive Product Planning adoption. 
266
    - Improvement - For Paid, continue making incremental, iterative progress towards [Plan's core JTBD](/direction/plan/#problem-and-jobs-to-be-done) as outlined in the [Issue Tracking direction](/direction/plan/issue_tracking/). 
267 268
  implementation:
    status: Complete
269
    reasons:
270 271
    - xMAU data models are now updated with the new counter that aggregates unique users across many different issue actions.
    - New [Plan xMAU](https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)) dashboard is live.
272
  metric_name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly 
273
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
274 275
    chart: 9967544
    dashboard: 758607
Jerome Ng's avatar
Jerome Ng committed
276 277
    embed: v2
  sisense_data_secondary:
Jerome Ng's avatar
Jerome Ng committed
278 279
    chart: 9980295
    dashboard: 758607
280
    embed: v2
281
  urls:
282
  - https://about.gitlab.com/direction/plan/issue_tracking/
283
  - https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)
Jerome Ng's avatar
Jerome Ng committed
284

Jerome Ng's avatar
Jerome Ng committed
285
- name: Plan:Product Planning - Paid GMAU - MAU creating Epics each Month
286 287
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users creating Epics each month.
288
  target: 3,900 Combined Unique Users for Self Managed and SaaS
289
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
290
  section: dev
Jerome Ng's avatar
Jerome Ng committed
291
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
292
  group: product_planning
293
  public: true
Jerome Ng's avatar
Jerome Ng committed
294
  pi_type: Paid GMAU
295
  product_analytics_type: Both
296 297 298 299 300
  is_primary: true
  is_key: false
  health:
    level: 3
    reasons:
Christen Dybenko's avatar
Christen Dybenko committed
301
    - Insight - GMAU is steadily rising with a known dip in August and Nov/Dec due to holidays. 
302
      [Monthly epic creation hit an all time high of +8500 in December](https://app.periscopedata.com/app/gitlab/602355/Portfolio-Management-G-SMAU-and-Feature-Success-Tracking?widget=9245316&udv=962108) and this was underscored 
Christen Dybenko's avatar
Christen Dybenko committed
303 304 305 306 307
      by a [jump in epic creation by Ultimate and Premium self managed customers](https://app.periscopedata.com/app/gitlab/602355/Product-Planning-G-SMAU-and-Feature-Success-Tracking?widget=7941113&udv=0) 
    - Improvement - [Epic Swimlanes](https://gitlab.com/groups/gitlab-org/-/epics/3352) shipped in 
      13.6 and we feel they be a significant driver of MAU growth for Product Planning nearterm (n2m) - giving 
      visibility to the feature on boards and adding additional value to better visualize your in-flight issues 
      through swimlanes.
308
    - Improvement - [Epic Boards](https://gitlab.com/groups/gitlab-org/-/epics/2864) is planned to ship in 13.9 and should continue the trend into the end of FY21 by enabling Portfolio-level planning.  
Jerome Ng's avatar
Jerome Ng committed
309 310
  implementation:
    status: Complete
311
    reasons:
312 313
    - Dashboarding - we've instrumented core metrics but are looking to expand the
      MAU definition to include all epic usage, not just creation, for a more accurate
314
      GMAU metric - [Issue](https://gitlab.com/gitlab-data/analytics/-/issues/5280). 
315
    - 2021-01 update - Similar to the Project Planning team, we have identified the [additional instrumentation](https://gitlab.com/gitlab-org/gitlab/-/issues/292253) needed to accomplish this a GMAU that includes all interactions with epics. 
316
  metric_name: epics 
317
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
318 319
    chart: 9980303
    dashboard: 758607
320 321
    embed: v2
  urls:
Christen Dybenko's avatar
Christen Dybenko committed
322
    
Mark Wood's avatar
Mark Wood committed
323
- name: Plan:Certify - Paid GMAU - MAU Interacting with Requirements
324 325 326 327 328
  base_path: "/handbook/product/performance-indicators/"
  definition: The number of unique users interacting (create, view, reference) each
    month
  target: TBD
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
329
  section: dev
Jerome Ng's avatar
Jerome Ng committed
330
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
331
  group: product_planning
332
  public: true
Jerome Ng's avatar
Jerome Ng committed
333
  pi_type: Paid GMAU
334
  product_analytics_type: Both
Jerome Ng's avatar
Jerome Ng committed
335
  is_primary: false
336 337
  is_key: false
  health:
Jerome Ng's avatar
Jerome Ng committed
338
    level: 0
339
    reasons:
Mark Wood's avatar
Mark Wood committed
340 341 342
    - Note - [Requirement interaction tracking](https://gitlab.com/gitlab-org/product-intelligence/-/issues/370) issue is complete. 
      Data is in the process of being aggregated. Early data shows healthy adoption of test report linking.
      [Tracking issue](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
343
    - Improvement - UX improvements for [Requirements Management](https://gitlab.com/groups/gitlab-org/-/epics/3708)
Mark Wood's avatar
Mark Wood committed
344
      based on completed research study, Requirement [Export MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/290813).
Jerome Ng's avatar
Jerome Ng committed
345 346
  implementation:
    status: Dashboard
347
    reasons:
Justin Farris's avatar
Justin Farris committed
348
    - Dashboarding - Requirement Creation is instrumented is complete. Tracking issue for gmau dashboard [here](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
349
  metric_name: TBD
Mark Wood's avatar
Mark Wood committed
350
- name: Plan:Certify - Other - Number of Requirements Created Monthly
351 352
  base_path: "/handbook/product/performance-indicators/"
  definition: A monthly count of the number of requirements created.
Mark Wood's avatar
Mark Wood committed
353
  target: 1000 by end of March 2021
354
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
355
  section: dev
Jerome Ng's avatar
Jerome Ng committed
356
  stage: plan
Jerome Ng's avatar
Jerome Ng committed
357
  group: product_planning
358
  public: true
Jerome Ng's avatar
Jerome Ng committed
359
  pi_type: Other
360
  product_analytics_type: Both
361 362 363 364 365
  is_primary: false
  is_key: false
  health:
    level: 3
    reasons:
Mark Wood's avatar
Mark Wood committed
366 367
    - Improvement - [Requirement Export MVC](https://gitlab.com/gitlab-org/gitlab/-/issues/290813) feature to release in 13.8
    - Insight - Requirement creation has decreased. Investigation shows a substantial set of users are Universities, which explains the downturn for the holidays. 
Mark Wood's avatar
Mark Wood committed
368
    - Insight - Self-managed instances creating requirements continues to grow.   
Jerome Ng's avatar
Jerome Ng committed
369 370
  implementation:
    status: Dashboard
371
    reasons:
372
    - Dashboarding -  accurate instrumentation on Self Managed, still needs verification
373
      for SaaS - [issue](https://gitlab.com/gitlab-data/analytics/-/issues/5270)
Jerome Ng's avatar
Jerome Ng committed
374
  metric_name: requirements_created
375
  sisense_data:
Mark Wood's avatar
Mark Wood committed
376
    chart: 9666808
377
    dashboard: 634539
378 379 380
    embed: v2
  urls:
  - https://gitlab.com/gitlab-data/analytics/-/issues/5270
381

James Ramsay's avatar
James Ramsay committed
382
- name: Create - SMAU - Unique users writing to a Git repository
383
  base_path: "/handbook/product/performance-indicators/"
384 385
  definition: A rolling count of the number of unique users who performed any Git write operation within the last 28 days.
  target: |
James Ramsay's avatar
James Ramsay committed
386
    Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
387
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
388
  section: dev
Jerome Ng's avatar
Jerome Ng committed
389
  stage: Create
390
  public: true
Jerome Ng's avatar
Jerome Ng committed
391
  pi_type: SMAU
392
  product_analytics_type: Both
393 394 395
  is_primary: true
  is_key: false
  health:
396
    level: 3
397
    reasons:
James Ramsay's avatar
James Ramsay committed
398 399 400 401 402
    - Observation - December 2020 SMAU is less than November 2020 SMAU. A
      similar drop can be observed in GMAU as well, suggesting a common cause.
      The cause is likely seasonality consistent with the end of year. For this
      reason January SMAU and GMAU is expected to return to November 2020
      levels and continue growing.
James Ramsay's avatar
James Ramsay committed
403 404 405 406 407 408
    - Insight - on the median GitLab instance, 33% of users perform a Git write
      operation each month
      ([source](https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=10435358&udv=1100338)).
    - Insight - on the median Paid GitLab instance (self-managed), 50% of users
      perform a Git write operation each month
      ([source](https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=10435361&udv=1100338)).
Jerome Ng's avatar
Jerome Ng committed
409 410
  implementation:
    status: Complete
411
    reasons:
James Ramsay's avatar
James Ramsay committed
412 413 414 415
    - Improvement - AMAU is currently over counted. A deduplicated counter was implemented
      [gitlab#234027](https://gitlab.com/gitlab-org/gitlab/-/issues/234027) in GitLab 13.6.
      The charts do not yet reflect the new counter
      [gitlab#291943](https://gitlab.com/gitlab-org/gitlab/-/issues/291943).
416 417
  metric_name: action_monthly_active_users_project_repo
  sisense_data:
Kathleen Tam's avatar
Kathleen Tam committed
418 419
    chart: 9967642
    dashboard: 758607
420
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
421 422 423 424
  sisense_data_secondary:
    chart: 9980442
    dashboard: 758607
    embed: v2
425
  urls:
Daniel Gruesso's avatar
Daniel Gruesso committed
426
  - https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard
427

James Ramsay's avatar
James Ramsay committed
428
- name: Create:Source Code - GMAU - Unique users writing to a project Git repository
429
  base_path: "/handbook/product/performance-indicators/"
430 431 432
  definition: A rolling count of the number of unique users who performed any Git write
    operation within the last 28 days
  target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
433
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
434
  section: dev
Jerome Ng's avatar
Jerome Ng committed
435 436
  stage: create
  group: source_code
437
  public: true
Jerome Ng's avatar
Jerome Ng committed
438
  pi_type: GMAU
439
  product_analytics_type: Both
440 441 442
  is_primary: true
  is_key: false
  health:
Daniel Gruesso's avatar
Daniel Gruesso committed
443
    level: 3
444
    reasons:
445
    - Starting in [GitLab 13.6](https://gitlab.com/gitlab-org/gitlab/-/issues/234027)
446
      AMAU for this metric takes into consideration all repository types.
447 448 449 450
    - Improvement - work towards providing users more insights while navigating their
      code with [Code Intelligence](https://gitlab.com/groups/gitlab-org/-/epics/1576)
    - Improvement - work towards providing more control over [managing repository
      size](https://gitlab.com/groups/gitlab-org/-/epics/3793)
Jerome Ng's avatar
Jerome Ng committed
451
  implementation:
452
    status: Complete
453
    reasons:
454
    - Enabled on GitLab.com
455 456
    - Released for self-managed in GitLab 13.3
  metric_name: action_monthly_active_users_project_repo
Daniel Gruesso's avatar
Daniel Gruesso committed
457
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
458 459 460 461 462 463
    chart: 9967642
    dashboard: 758607
    embed: v2
  sisense_data_secondary:
    chart: 9980442
    dashboard: 758607
Daniel Gruesso's avatar
Daniel Gruesso committed
464
    embed: v2
465
  urls:
Daniel Gruesso's avatar
Daniel Gruesso committed
466
  - https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=9520571&udv=0
James Ramsay's avatar
James Ramsay committed
467
  - https://gitlab.com/gitlab-org/gitlab/-/issues/234027
468

469
- name: Create:Source Code Secondary - GMAU - Unique users approving merge requests
470
  base_path: "/handbook/product/performance-indicators/"
471
  definition: A rolling count of the number of unique users who approved any merge request within the last 28 days
472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: source_code
  public: true
  pi_type: GMAU
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Not currently instrumented
    - Currently planning instrumentation and evaluating if this is a more relevant metric for the Source Code group
  urls:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/293014

489
- name: Create:Source Code Tertiary - GMAU - Unique users pushes rejected by push rules
490
  base_path: "/handbook/product/performance-indicators/"
491
  definition: A rolling count of the number of unique users who had one or more push rejected by any push rule within the last 28 days
492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: source_code
  public: true
  pi_type: GMAU
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Not currently instrumented
    - Currently planning instrumentation and evaluating if this is a more relevant metric for the Source Code group
  urls:
  - https://gitlab.com/gitlab-org/gitlab/-/issues/293016

James Ramsay's avatar
James Ramsay committed
509
- name: Create:Code Review - GMAU - Unique users using Merge Requests
510
  base_path: "/handbook/product/performance-indicators/"
511 512
  definition: This is defined as users who perform one or more of the following actions in
   a merge request - create, merge, or close a Merge Request.
Kai Armstrong's avatar
Kai Armstrong committed
513
  target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU. 
514
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
515
  section: dev
Jerome Ng's avatar
Jerome Ng committed
516
  stage: create
517
  group: code_review
518
  public: true
James Ramsay's avatar
James Ramsay committed
519
  pi_type: GMAU
520
  product_analytics_type: Both
521
  is_primary: true
522 523
  is_key: false
  health:
Kai Armstrong's avatar
Kai Armstrong committed
524
    level: 2
525
    reasons:
Kai Armstrong's avatar
Kai Armstrong committed
526 527 528
    - Target - This aligns with our current understanding that Code Review is about 50% of the MAU of the stage metric and continuing to maintain that difference.
    - Insight - Continued investigation in to the drop-off between SCM and Code Review is being investigated by the data team in [data issue 7468](https://gitlab.com/gitlab-data/analytics/-/issues/7468).
    - Insight - Paid users of Code Review make up about one-third of total Code Review users.
529 530
    - Improvement - Support for [custom commit messages when applying a suggested change](https://gitlab.com/gitlab-org/gitlab/-/issues/25381) will allow more teams with strict commit message guidelines to adopt this feature and maintain conformity with the project.
    - Improvement - Add [reviewers to merge request](https://gitlab.com/groups/gitlab-org/-/epics/1823) to better signal who is responsible for reviewing compared to being assigned without a clear action to take.
Jerome Ng's avatar
Jerome Ng committed
531
  implementation:
532
    status: Complete
533
    reasons:
534 535
    - Enabled on GitLab.com 2020-07-04
    - Enabled for self-managed on GitLab 13.2
536
    - Working on more comprehensive MAU tracking in [&4948](https://gitlab.com/groups/gitlab-org/-/epics/4948)
537
  metric_name: merge_requests_users 
Daniel Gruesso's avatar
Daniel Gruesso committed
538
  sisense_data:
539 540
    chart: 10426645
    dashboard: 786738
Daniel Gruesso's avatar
Daniel Gruesso committed
541
    embed: v2
542
  sisense_data_secondary:
543
    chart: 9601815
Kai Armstrong's avatar
Kai Armstrong committed
544
    dashboard: 10624715
545
    embed: v2
546
  urls:
547
  - https://app.periscopedata.com/app/gitlab/526749/Create---Code-Review-category-dashboard
James Ramsay's avatar
James Ramsay committed
548

Kai Armstrong's avatar
Kai Armstrong committed
549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573
- name: Create:Code Review - Other - % of MAU using Merge Requests
  base_page: "/handbook/product/performance-indicators/"
  definition: 
  target: TBD - Primarily this metric is used for monitoring adoption on a broad scale and remains consistent.
  org: Dev Section
  section: dev
  stage: create
  group: code_review
  public: true
  pi_type: Other
  product_analytics_type: Both
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Target - this metric is primarily monitored to remain at a consistent level.
  implementation:
    status: Complete
  metric_name: merge_requests_users
  sisense_data:
    chart: 9601815
    dashboard: 664109
    embed: v2

574
- name: Create:Gitaly - Other - Availability
575
  base_path: "/handbook/product/performance-indicators/"
576 577 578 579 580 581 582
  definition: Percentage of time during which Gitaly is fully operational and providing
    service to users within SLO parameters. Availability is measured according by
    Apdex (latency) and error rate. Gitaly is considered available when at least 50%
    of users are experiencing satisfactory latency, _and_ 50% of requests are completing
    successfully. If either of these conditions is not met, the service is experiencing
    an outage.
  target: 99.95%, consistent with GitLab.com availability service level objectives
583
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
584
  section: dev
585 586
  stage: create
  group: gitaly
587
  public: true
Jeremy Watson (ex-GitLab)'s avatar
Jeremy Watson (ex-GitLab) committed
588
  pi_type: Other
589
  product_analytics_type: Both
590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617
  is_primary: true
  is_key: false
  health:
    level: 0
    reasons:
    - Instrumentation for this metric started in 13.4 and is still ongoing.
  implementation:
    status: Instrumentation
    reasons:
    - Rolling 7 day metrics available in [Grafana](https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview)
      for GitLab.com. Current availability per Adpex is 99.98%.
    - Improvement - self-managed availability metrics are [planned](https://gitlab.com/gitlab-org/gitlab/-/issues/216683)
  metric_name: gitaly_apdex
  urls:
  - https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
  - https://gitlab.com/gitlab-org/gitlab/-/issues/216683

- name: Create:Gitaly - Other - Unique instances with Gitaly Cluster enabled
  base_path: "/handbook/product/performance-indicators/"
  definition: Number of recorded GitLab instances with Gitaly Cluster enabled
  target: TBD
  org: Dev Section
  section: dev
  stage: create
  group: gitaly
  public: true
  pi_type: Other
  product_analytics_type: Self-Managed
Christen Dybenko's avatar
Christen Dybenko committed
618
  is_primary: false
619 620
  is_key: false
  health:
621
    level: 2
622
    reasons:
623 624
    - No target defined as the feature is still being rolled out.
    - Improvement - introduce [Variable Replication Factor and auto-reblancing within a cluster](https://gitlab.com/groups/gitlab-org/-/epics/3372) to enable more enterprise-level adoption.
Jerome Ng's avatar
Jerome Ng committed
625
  implementation:
626
    status: Complete
627
    reasons:
Mark Wood's avatar
Mark Wood committed
628
    - Insight - Reference architecture has been updated with recommendations for Gitaly Cluster for 13.8.
629 630
    - Instrumentation for reporting on instances with Gitaly Cluster enabled was introduced in 13.0 with the GA of Gitaly Cluster.
  metric_name: gitaly_clusters
Jerome Ng's avatar
Jerome Ng committed
631
  sisense_data:
632 633
    chart: 9504807
    dashboard: 728407
Jerome Ng's avatar
Jerome Ng committed
634
    embed: v2
635
  urls:
636
  - https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
James Ramsay's avatar
James Ramsay committed
637 638

- name: Create:Editor - GMAU - Unique users committing via any web editor
639 640
  base_path: "/handbook/product/performance-indicators/"
  definition: A rolling count of the number of unique users who performed a Git write
641 642
    operation to any respository using the Web IDE, Web Editor or Snippet within a
    stage in the last 28 days
Eric Schurter's avatar
Eric Schurter committed
643
  target: 4% growth month over month for All 
644
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
645
  section: dev
Jerome Ng's avatar
Jerome Ng committed
646 647
  stage: create
  group: editor
648
  public: true
Jerome Ng's avatar
Jerome Ng committed
649
  pi_type: GMAU
650
  product_analytics_type: Both
651
  is_primary: true
652 653
  is_key: false
  health:
654
    level: 0
655
    reasons:
656
    - New target GMAU to be set once we get cumulative, de-duped data from newly aligned categories.
Eric Schurter's avatar
Eric Schurter committed
657
    - Insight - Slight decrease in write operations in December, consistent with the overall decrease in write operations to a git repository. No year-over-year data but this is likely due to the holiday season.
Jerome Ng's avatar
Jerome Ng committed
658
  implementation:
Jerome Ng's avatar
Jerome Ng committed
659
    status: Complete
660
    reasons:
661
    - Instrumentation of [Static Site Editor and Wiki to roll up into GMAU](https://gitlab.com/gitlab-org/gitlab/-/issues/283947) completed in 13.7, waiting for data to be reflected in the charts.
662
  metric_name: action_monthly_active_users_ide_edit 
663
  sisense_data:
Jerome Ng's avatar
Jerome Ng committed
664 665
    chart: 9967737
    dashboard: 758607
666
    embed: v2
Jerome Ng's avatar
Jerome Ng committed
667 668 669
  sisense_data_secondary:
    chart: 9980340
    dashboard: 758607
Jerome Ng's avatar
Jerome Ng committed
670
    embed: v2
671 672
  urls:
  - https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU
James Ramsay's avatar
James Ramsay committed
673 674

- name: Create:Editor - AMAU - Unique users committing by web editor
675 676 677 678 679 680
  base_path: "/handbook/product/performance-indicators/"
  definition: A rolling count of the number of unique users who performed a Git write
    operation to any respository using the Web IDE, Web Editor or Snippet within a
    stage in the last 28 days - users grouped by editor used
  target: TBD
  org: Dev Section
Jerome Ng's avatar
Jerome Ng committed
681
  section: dev
682 683 684
  stage: create
  group: editor
  public: true
James Ramsay's avatar
James Ramsay committed
685
  pi_type: AMAU
686
  product_analytics_type: Both
687 688 689 690 691 692
  is_primary: false
  is_key: false
  health:
    level: 0
    reasons:
    - Data collection is beginning, but it's too early to define a target as instances continue to upgrade and report data.
Eric Schurter's avatar
Eric Schurter committed
693
    - Insight - Editing through the Web Editor (single file editor) is 2-3 to 1 for users of the Web IDE. This confirms data recognized previously based on navigation paths that the single file editor is more widely used. We need to continute monitoring this to determine future investments in the Web IDE.
694
    - Note - Important to call out that these are not unique users, so the same user can be reported as using multiple web editors. 
Jerome Ng's avatar
Jerome Ng committed
695
  implementation:
Eric Schurter's avatar
Eric Schurter committed
696
    status: Completed [Dashboard](https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU)
697
    reasons:
698
    - Instrumentation is in place for both self-managed and SaaS.
Jerome Ng's avatar
Jerome Ng committed
699
    - Updating dashboards to for Paid / All breakouts
700
  metric_name: action_monthly_active_users_web_ide_edit, action_monthly_active_users_sfe_edit, action_monthly_active_users_snippet_editor_edit
701 702 703 704
  sisense_data:
    chart: 9788102
    dashboard: 741875
    embed: v2
705
  urls:
706
  - https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU
James Ramsay's avatar
James Ramsay committed
707