[Experiment Tracking] ci_syntax_templates - Empty versatile template for CI - Hello World
What
Track the status of an experiment through to removal.
- Experiment key:
ci_syntax_templates
- Feature flag name:
ci_syntax_templates_experiment_percentage
This is an experiment tracking issue for: #248 (closed) using the scoped experiment label.
As well as defining the experiment rollout and cleanup, this issue incorporates the relevant
Feature Flag Roll Out
steps.
Owners
- Team:
group::activation
- Most appropriate slack channel to reach out to:
#g_activation
- Best individual to reach out to: @alexbuijs
Expectations
Tracking
Tracking takes place in the database in our experiment_users
table. A record is inserted whenever a user clicks on the Setup CI/CD
button. The user is considered 'converted' when a pipeline is created by the user.
What are we expecting to happen?
More converted users for users in the experimental_group
than for users in the control_group
.
What might happen if this goes wrong?
What can we monitor to detect problems with this?
Staging Test
To force this experiment on staging go to an editable project and append -/new/master?commit_message=Add+.gitlab-ci.yml&file_name=.gitlab-ci.yml&force_experiment=ci_syntax_templates
to the URL.
Rollout plan:
- Runtime: 30 days or until we reach statistical significance
- We will roll this out behind a feature flag and expose this to 20% of users to start then ramp it up from there.
- feature flag based on experiment key
<experiment-key>
(seeexperimentation.rb
in GitLab, append '_experiment_percentage')
/chatops run feature set ci_syntax_templates_experiment_percentage 50
Roll Out Steps
-
Confirm that QA tests pass with the feature flag enabled (if you're unsure how, contact the relevant stable counterpart in the Quality department) -
Enable on staging ( /chatops run feature set feature_name true --staging
) -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab feature_name true
) -
Coordinate a time to enable the flag with the SRE oncall and release managers - In
#production
mention@sre-oncall
and@release-managers
. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
- In
-
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
(/chatops run feature set feature_name true
) -
Cross post chatops Slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled
-
Remove experiment code and feature flag and add changelog entry - a separate cleanup issue might be required
-
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel