[Experiment Rollout] invite_members_for_task - Invite members for task

What

Track the status of an experiment through to removal.

  1. Experiment key: invite_members_for_task
  2. Framework: gitlab_experiment
  3. Feature flag name: invite_members_for_task

This is an experiment tracking issue for: https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/432 using the scoped experiment label.

As well as defining the experiment rollout and cleanup, this issue incorporates the relevant Feature Flag Roll Out steps.

Owners

  • Team: group::activation
  • Most appropriate slack channel to reach out to: #g_activation
  • Best individual to reach out to: @alexbuijs

Expectations

What are we expecting to happen?

What might happen if this goes wrong?

What can we monitor to detect problems with this?

Tracked data

experiments table
id name
experiment_id invite_members_for_task
experiment_subjects table (note: we do not distinguish between the 3 variants in the database)
experiment_id namespace_id variant
experiment_id namespace_id 0 (control) or 1 (candidate)
onboarding_progresses table
namespace_id git_write_at (task 0) repository_imported_at (task 0) pipeline_created_at (task 1) issue_created_at (task 2)
namespace_id NOT NULL (converted) or NULL (not converted) NOT NULL (converted) or NULL (not converted) NOT NULL (converted) or NULL (not converted) NOT NULL (converted) or NULL (not converted)

Snowplow tracking

The following events are tracked anonymously through Snowplow:

activity category action context label property sent from variants
Link clicked in the email groups:show modal_opened_from_email n/a n/a gitlab_experiment json schema frontend candidate
Link clicked in the email groups:show submit selected_tasks_to_be_done A string with any combination of ci, code, and issues, separated by a comma gitlab_experiment json schema frontend candidate

Staging Test

Experiment tracking log

Experiment Results

Roll Out Steps

  • Confirm that QA tests pass with the feature flag enabled (if you're unsure how, contact the relevant stable counterpart in the Quality department)
  • Enable on staging (/chatops run feature set invite_members_for_task 50 --actors --staging)
  • Test on staging
  • Ensure that documentation has been updated
  • Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab feature_name true)
  • Coordinate a time to enable the flag with the SRE oncall and release managers
    • In #production mention @sre-oncall and @release-managers. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
  • Announce on the issue an estimated time this will be enabled on GitLab.com
  • Enable on GitLab.com by running chatops command in #production (/chatops run feature set invite_members_for_task 50 --actors)
  • Cross post chatops Slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
  • Announce on the issue that the flag has been enabled
  • Remove experiment code and feature flag and add changelog entry - a separate cleanup issue might be required
  • After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel
  • Assign to the product manager to update the knowledge base (if applicable)

Rollback Steps

  • This feature can be disabled by running the following Chatops command:
/chatops run feature delete invite_members_for_task
Edited by Alex Buijs