Skip to content

Review experiment cadence metric calculation and Growth product knowledge base to ensure consistency in calculation and reporting

Problem

In Q2 Growth engineering introduced a Growth development metric Growth experiment cadence to measure experiments per engineer per month https://about.gitlab.com/handbook/engineering/development/growth/#experiment-cadence

Growth product managers maintain a knowledge base of experiments (planned, active, and concluded) on https://about.gitlab.com/direction/growth/#growth-experiments-knowledge-base

Let's compare and

  • Update the experiment knowledge base
  • Review and ensure the experiment cadence metric accurately reflects

For example, the cadence measures new experiments based on experiment feature flags being added to the codebase. In most cases when a change is made to an existing experiment this will not be counted as a new experiment. Often, this does not include variants or changes added to existing experiments. We need to ensure we are aligned on this and make process or reporting changes if necessary.

Steps

Confirm experiments knowledge base is up to date - this should list one row for each experiment run (at least in 2021)

https://about.gitlab.com/direction/growth/#growth-experiments-knowledge-base

Review knowledge base to confirm it is up to date with experiment feature flag status

Comparison based on Growth Experiment Knowledge base as at 2021-09-20

https://docs.google.com/spreadsheets/d/1wMCqx50YH5yPIVIGS5d-mzhwP_-Svr_6F3z8LXpD3WQ/edit?usp=sharing (GitLab internal)

Summary: 21 experiments listed in the knowledge base require updates

Walkthrough explaining how the spreadsheet was updated: https://www.youtube.com/watch?v=pAf0vwMIqHM

Please open a merge requests to update https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/source/direction/growth/index.html.md

Links:

Edited by Phil Calder