[Experiment Rollout] code_quality_walkthrough - Introduce users to verify via code quality template walkthrough
What
Track the status of an experiment through to removal.
- Experiment key:
code_quality_walkthrough
- Framework:
gitlab_experiment
- Feature flag name:
code_quality_walkthrough
This is an experiment tracking issue for: https://gitlab.com/gitlab-org/growth/engineering/-/issues/5436
using the scoped experiment label.
As well as defining the experiment rollout and cleanup, this issue incorporates the relevant
Feature Flag Roll Out
steps.
Owners
- Team:
group::activation
- Most appropriate slack channel to reach out to:
#g_activation
- Best individual to reach out to: @alexbuijs
Expectations
What are we expecting to happen?
What might happen if this goes wrong?
What can we monitor to detect problems with this?
Tracked data
Note: you can utilize CXL calculator to determine if your experiment has reached signifigance, it also includes an estimate for how much longer an experiment will need to run for before reaching signifigance.
Staging Test
Experiment tracking log
- 2021-06-30 11:41 UTC - initial rollout to 50% of actors
The experiment is enabled on a namespace level, for namespaces created less than 90 days ago, for users who are able to create pipelines and for projects that do not already have a committed .gitlab-ci.yml
file.
Database tracking
The success rate of the experiment can be determined by joining the experiments
, experiment_subjects
and onboarding_progresses
tables:
experiments
table
id | name |
---|---|
experiment_id | code_quality_walkthrough |
experiment_subjects
table
experiment_id | group_id | variant |
---|---|---|
experiment_id | group_id |
0 (control) or 1 (candidate) |
onboarding_progresses
table
namespace_id | pipeline_created_at |
---|---|
group_id |
NOT NULL (converted) or NULL (not converted) |
Snowplow tracking
The following events are tracked through Snowplow:
activity | category | action | context | sent from | groups |
---|---|---|---|---|---|
Visited a projects' empty pipelines page | code_quality_walkthrough |
assignment |
json schema | Backend | experiment |
"Get started with CI/CD" or "Add a code quality job" button clicked | code_quality_walkthrough |
cta_clicked |
json schema | Frontend | control + experiment |
Commit Popover displayed | code_quality_walkthrough |
commit_ci_file_displayed |
json schema | Frontend | experiment |
Commit Popover "Got it" link clicked | code_quality_walkthrough |
commit_ci_file_dismissed |
json schema | Frontend | experiment |
.gitlab-ci.yml file committed |
code_quality_walkthrough |
commit_created |
json schema | Backend | experiment |
Running Pipeline Popover displayed | code_quality_walkthrough |
running_pipeline_displayed |
json schema | Frontend | experiment |
Running Pipeline Popover "Got it" link clicked | code_quality_walkthrough |
running_pipeline_dismissed |
json schema | Frontend | experiment |
Success Pipeline Popover displayed | code_quality_walkthrough |
success_pipeline_displayed |
json schema | Frontend | experiment |
Success Pipeline Popover "View the logs" link clicked | code_quality_walkthrough |
success_pipeline_view_logs |
json schema | Frontend | experiment |
Failed Pipeline Popover displayed | code_quality_walkthrough |
failed_pipeline_displayed |
json schema | Frontend | experiment |
Failed Pipeline Popover "View the logs" link clicked | code_quality_walkthrough |
failed_pipeline_view_logs |
json schema | Frontend | experiment |
Troubleshoot Job Alert displayed | code_quality_walkthrough |
troubleshoot_job_displayed |
json schema | Frontend | experiment |
Troubleshoot Job Alert dismissed | code_quality_walkthrough |
troubleshoot_job_dismissed |
json schema | Frontend | experiment |
Troubleshoot Job Alert "Read the documentation" button clicked | code_quality_walkthrough |
troubleshoot_job_clicked |
json schema | Frontend | experiment |
Experiment Results
Roll Out Steps
-
Confirm that QA tests pass with the feature flag enabled (if you're unsure how, contact the relevant stable counterpart in the Quality department) -
Enable on staging ( /chatops run feature set feature_name true --staging
) -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab feature_name true
) -
Coordinate a time to enable the flag with the SRE oncall and release managers - In
#production
mention@sre-oncall
and@release-managers
. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
- In
-
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
(/chatops run feature set feature_name true
) -
Cross post chatops Slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove experiment code and feature flag and add changelog entry - a separate cleanup issue might be required -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel -
Assign to the product manager to update the knowledge base (if applicable)
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set code_quality_walkthrough false