Feature Instrumentation for Past and Upcoming Features

Problem

Analysis conducted by the CSM team reveals that only 50% of our premium features are currently instrumented. While we released on an average of 30 features per milestone between versions 16.6 and 16.9, we observed that, on average, only 6 features are instrumented per milestone.

The absence or delay of instrumentation presents several challenges:

  1. Missed Insights: Without instrumentation, we're missing out on valuable insights that could be derived from the data.
  2. Deferred Instrumentation: When instrumentation is delayed to future milestones, self-managed customers must upgrade to the instrumented version for us to gather insights. This delay means we have less data from self-managed sources.
  3. Hindered Customer Support: Lack of instrumentation impedes the ability of our CSM team to assist customers seeking to understand feature usage and adoption within their organisations.

(above cited from issue 452133)

Currently we don't have a clearly defined process that ensures engineering and product implement product instrumentation prior to launch. We also need to be able to retroactively evaluate what features have been instrumented and need instrumentation from past feature launches.

Desired Outcome

Our goal is to increase product instrumentation across our features to deliver greater product insights and minimize deferred instrumentation.

For past feature launches: Audit and analysis of our current product instrumentation. To better understand where we need to go and to be able to make attainable goals, we need to assess where we currently stand and set objectives for improvement.

For future feature launches: To enable product teams to instrument proactively, we will be creating process change to reinforce positive behavior and best practices with product instrumentation. This will help us minimize retroactively adding product instrumentation and instead further reinforce moving this process prior to launch.

Next Steps

Previously launched features:

  • Product Data Insights team to task PMs at a Category level to asses instrumentation with analyst

  • Trial through Q2 with two PMs. Mark in category.yaml with below markers

    Instrumentation complete and satisfactory

    💛 Some instrumentation, but not complete

    🔴 No instrumentation - instrumentation needed

    🔶 Instrumentation not needed

  • Gather feedback from product data insights team + product manager

  • Further investigate which product divisions are highest priority for post instrumentation

  • Create Analytics Instrumentation dashboard with status (example here [(https://10az.online.tableau.com/#/site/gitlab/workbooks/2437638/views)\])

Upcoming features:

  • Create an draft to update product development flow to add in product instrumentation in plan
  • Create MR for Handbook
  • Once MR is created; analytics instrumentation team to follow standard processes and procedures in combination with newly created documentation
  • Pilot on 2-3 stages then gather feedback + iterate
Edited by Michaela Seferian-Jenkins