Skip to content

GitLab Next

Closed
Opened by Hila Qu@hilaquDeveloper

[Proposal] GitLab Experiment Process

Context

This issue is to propose a process to design, execute and launch experiment at GitLab

  • Experiment template: experiment design doc and add results after testing
  • Experiment design repo: central location for all experiment design docs

Current resources

  1. process https://about.gitlab.com/handbook/engineering/development/growth/#running-experiments
  2. experiment ideation template https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/new?issuable_template=Growth%20experiment
  3. experiment backlog (ideas) https://gitlab.com/groups/gitlab-org/-/boards/2028884
  4. experiment tracking (status) template https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/new?issuable_template=experiment_tracking_template
  5. experiment tracking board https://gitlab.com/groups/gitlab-org/-/boards/1352542?label_name[]=devops%3A%3Agrowth&label_name[]=growth%20experiment&label_name[]=experiment%20tracking

Related issues

  1. Experiment cleanup experimentation#6 (closed)
  2. Publishing results experimentation#7 (closed)

Proposed Process

  1. Generate experiment idea backlog
  • Based on the OKR, KPI, focus areas
  • Backed up by data: quantitative & qualitative
  1. Fill an experiment design template for the experiment you want to test
  1. In the experiment design, @mention to growth PMs to get feedback
  • Note this is not meant to be a blocker from proceeding, but as a way to increase experiment quality

  1. Create an epic in your team board and copy key contents from the experiment design there & create issues for dev, design and link to the epic

  2. Build and launch experiment

  3. Conclude experiment once statistical significance is reached or it is decided the experiment variants are indifferent from control

  4. Analyze results, summary learning and go back to the experiment doc to fill it there - so that we completed

  5. Decide the next steps for the experiment: 1) Continue to iterate 2)Roll out to 100%; 3)Kill and clean the code

Flow chart

Growth_Experiment_Flow_Chart_v1.2__5_.pdf

Edited by Phil Calder