Feature instrumentation reminder - release 16.11

Problem

Analysis conducted by the CSM team reveals that only 50% of our premium features are currently instrumented. While we released on an average of 30 features per milestone between versions 16.6 and 16.9, we observed that, on average, only 6 features are instrumented per milestone.

The absence or delay of instrumentation presents several challenges:

  1. Missed Insights: Without instrumentation, we're missing out on valuable insights that could be derived from the data.
  2. Deferred Instrumentation: When instrumentation is delayed to future milestones, self-managed customers must upgrade to the instrumented version for us to gather insights. This delay means we have less data from self-managed sources.
  3. Hindered Customer Support: Lack of instrumentation impedes the ability of our CSM team to assist customers seeking to understand feature usage and adoption within their organisations.

Goal

The primary objective of this issue is to track features released in each milestone and gain insights from Product Managers and Engineering Managers regarding the current status of instrumentation for these features. By doing so, we aim to identify any obstacles hindering instrumentation and collaborate with teams to address these gaps effectively. Our overarching aim is to increase the number of instrumented features at the time of release, ensuring the availability of timely data essential for guiding data-driven decisions.

Furthermore, by highlighting the number of metrics being instrumented across different stages in each milestone, we aim to inspire more teams to kickstart instrumentation efforts.

Product Manager tasks

  • Please review the feature list for your stage and category and fill in the Instrumentation and Additional Information columns. Here are further details:

For the Instrumentation column, please select one of the following options:

  1. Instrumentation added
  2. Planned for X milestone
  3. Not Planned
  4. Other

For the Additional information column,

  • If you have added instrumentation, please provide the link to the instrumentation issue or MR. This will help us link features to their respective events/metrics and maintaining a single source of truth for feature-to-event/metrics mapping. Currently, this process is manual, but we are actively working on automating it. You won't need to add this information in the future.
  • If you choose any of the other options, please provide additional information that will help us understand any blockers or pain points regarding feature instrumentation.

Feature list

Table contains list of features that made a release post in 16.11 and is sorted by stage and category. Additional details for the features available here

Stage Category Feature Reporter Instrumentation Additional Information
ai-powered Duo Chat GitLab Duo Chat now generally available @tlinz DAU, MAU, WAU, Retention What's missing: additional metrics and more details about SM usage/users
create Editor Extensions GitLab Duo Chat available in JetBrains IDEs @dashaadu
create Code Review Workflow, Team Planning Group comment templates @phikai
data_stores Groups Projects Sidebar for metadata on the project overview page @mle
data_stores Groups Projects Updated project archiving functionality @lohrc Not planned These were UX changes that don't make sense to be instrumented. The quarterly SUS, etc. surveys are probably better suited to measure impact of these improvements.
deploy Auto DevOps Build step of Auto DevOps upgraded @nagyv-gitlab
govern Security Policy Management Extend policy bot comment with violation data @g.hickman
govern System Access Webhook notifications for expiring group and project access tokens @hsutor
govern User Management More username options @hsutor
govern System Access Authenticate to Google Cloud with workload identity federation @hsutor
govern User Management Disable personal access tokens for Enterprise Users @hsutor
govern System Access Users list search and filter improvements @hsutor
govern System Access Renew application secret with API @hsutor
govern Security Policy Management Issue with duplicate security policies resolved @g.hickman Not planned This is really more of a bug fix / feature enhancement
govern Security Policy Management Security policy scopes @g.hickman
govern Compliance Management Display linked Security Policies in Compliance Frameworks @g.hickman
manage Webhooks Custom webhook headers @m_frankiewicz
manage Webhooks Test project hooks with the REST API @m_frankiewicz
manage Integrations GitLab for Slack app configurable for groups and instances @m_frankiewicz
manage Importers Configurable import jobs limit @m_frankiewicz
monitor Product Analytics Visualization Understand your users better with Product Analytics @lfarina8 Instrumentation added
monitor Product Analytics Visualization Explore your Product Analytics data with GitLab Duo @lfarina8 Instrumentation added
package Container Registry Connect Google Artifact Registry to your GitLab project @trizzi
plan Portfolio Management Visually distinguish epics using colors @amandarueda
plan Value Stream Management Value stream events can now be calculated cumulatively @hsnir1
plan Wiki Autocomplete support for links to wiki pages @mmacfarlane
plan Pages Improved GitLab Pages visibility in sidebar @mmacfarlane
platforms GitLab Dedicated, Switchboard Email notifications for changes made using Switchboard @lbortins
secure Software Composition Analysis Dependency graph support for dependency scanning SBOMs @johncrowley
secure Software Composition Analysis Dependency Scanning support for Yarn v4 @johncrowley
secure DAST DAST analyzer performance updates @smeadzinger Instrumentation added team flame chart
verify Secrets Management Expanded Hashicorp Vault Secrets support, including Artifactory and AWS @jocelynjane
verify Pipeline Composition Improve automatic retry for failed CI jobs with specific exit codes @dhershkovitch
verify GitLab Runner Core GitLab Runner 16.11 @deastman
verify Automate the creation of Google Compute Engine Runners from GitLab - Public Beta @DarrenEastman
verify Continuous Integration CI Option to cancel a pipeline immediately if any jobs fails @rutshah
verify Pipeline Composition Control who can download job artifacts @dhershkovitch
verify Pipeline Composition Improved pipeline details page @dhershkovitch

Instrumentation wins this milestone

image.pngLink to Tableau dashboard

Latest on instrumentation

We are looking to pair with teams to help drive instrumentation. More details in this slack announcement

cc @stkerr @bastirehm

Edited by Torsten Linz