Skip to content

[Experiment Rollout] Create a marketing header when user is on GitLab and not logged in

Summary

This issue tracks the rollout and status of an experiment through to removal.

  1. Experiment key / feature flag name: logged_out_marketing_header
  2. Epic or issue link: #346438 (closed)

✅ Dependency: https://gitlab.com/gitlab-data/analytics/-/issues/11690

This is an experiment rollout issue using the scoped experiment label. As well as defining the experiment rollout and cleanup, this issue incorporates the relevant Feature Flag Roll Out steps.

Owners

  • Team: group::activation
  • Most appropriate slack channel to reach out to: #g_activation
  • Best individual to reach out to: @eugielimpin
  • Product manager (PM): @jstava

Stakeholders

Expectations

What are we expecting to happen?

What might happen if this goes wrong?

What can we monitor to detect problems with this?

Tracked data

Snoplow-tracked events and the relevant payloads:

  1. Assignment: user sees the marketing header 
    "variant":"candidate",
"key":"e1181ef4e67173064bf8b55e095e69ae",
"se_action":"assignment"
  2. User clicks Sign up now button in the header 
    "key":"e1181ef4e67173064bf8b55e095e69ae",
"variant":"candidate",
"se_action":"click_button",
"se_label":"sign_up_now"
  3. User signs up 
    "user_id":103,
"namespace_id":null,
"project_id":null,
"variant":"candidate",
"key":"e3129f6d7ecda8eea397b376d07b6829",
"migration_keys":[ "e1181ef4e67173064bf8b55e095e69ae" ],
"se_action":"signed_up"
  4. Namespace is created by the user 
    "user_id":103,
"namespace_id":195,
"variant":"candidate",
"key":"e3129f6d7ecda8eea397b376d07b6829",
"se_action":"namespace_created"

migration_keys are sent during signed_up event and that's how we can connect the namespace_created event and the earlier events (assignment, click_button sign_up_now).

Note: you can use the CXL calculator to determine if your experiment has reached significance. The calculator includes an estimate for how much longer an experiment must run for before reaching significance.

Rollout plan

  • Runtime in days, or until we expect to reach statistical significance: 30
  • We will roll this out behind a feature flag and expose this to 50% of actors to start then ramp it up from there.

/chatops run feature set logged_out_marketing_header 50 --actors

Status

Preferred workflow

The issue should be assigned to the Product manager (PM) or Engineer (Eng) as follows:

  1. PM determines and manages the status of the experiment (assign this issue to the PM)
  2. PM asks for initial rollout on production, or changes to the status (assign to an Eng)
  3. Eng changes the status using chatops (reassign to the PM)
  4. When concluded, PM updates the 'Roll Out Steps' and adds a milestone (assigns to an Eng)

The current status and history can be viewed using the:

In this rollout issue, ensure the scoped experiment:: label is kept accurate.

Experiment Results

Roll Out Steps

  • Confirm that end-to-end tests pass with the feature flag enabled. If there are failing tests, contact the relevant stable counterpart in the Quality department to collaborate in updating the tests or confirming that the failing tests are not caused by the changes behind the enabled feature flag.
  • Enable on staging (/chatops run feature set logged_out_marketing_header true --staging)
  • Test on staging
  • Ensure that documentation has been updated
  • Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab feature_name true)
  • Coordinate a time to enable the flag with the SRE oncall and release managers
    • In #production mention @sre-oncall and @release-managers. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
  • Announce on the issue an estimated time this will be enabled on GitLab.com
  • Enable on GitLab.com by running chatops command in #production (/chatops run feature set feature_name true)
  • Cross post chatops Slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
  • Announce on the issue that the flag has been enabled
  • Remove experiment code and feature flag and add changelog entry - a separate cleanup issue might be required
  • After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel
  • Assign to the product manager to update the knowledge base (if applicable)

Rollback Steps

  • This feature can be disabled by running the following Chatops command:
/chatops run feature set logged_out_marketing_header false

Experiment Successful Cleanup Concerns

  • Several people mentioned that the height of the header looks off, like a visual bug. We need to fix this if the experiment is successful.
  • Becomes more of a problem on some viewports. We also need to check mobile. Text items wrap.

image

image

Items to be considered if candidate experience is to become a permanent part of GitLab

Edited by Jacki Bauer