[Experiment Rollout] ordered_integrations - Order integrations by popularity

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Label this issue

Summary

This issue tracks the rollout and status of an experiment through to removal.

1. Feature flag name: ordered_integrations
2. Epic or issue link: https://gitlab.com/gitlab-org/gitlab/-/issues/584797

This is an experiment rollout issue using the scoped experiment label. As well as defining the experiment rollout and cleanup, this issue incorporates the relevant Feature Flag Roll Out steps.

Owners

• Team: group::acquisition
• Most appropriate slack channel to reach out to: #g_acquisition
• Best individual to reach out to: @davidhampgonsalves
• Product manager (PM): @jtucker_gl

Stakeholders

• PM: @jtucker_gl
• Group: group::acquisition

Expectations

What are we expecting to happen?

Ordering integrations by popularity (based on Snowflake query results) on the group integrations page will improve discoverability and lead to higher integration adoption rates compared to the current alphabetical sort order.

What might happen if this goes wrong?

The integrations page could display an unexpected order. No data loss risk — this is a display-only change. Roll back by disabling the feature flag.

What can we monitor to detect problems with this?

Tracked data

Snowplow structured events for viewing the integrations page (must include the gitlab_experiment context to appear in fct_behavior_structured_event_experiment).

Results can be analyzed via the GLEX Experiment Analysis Dashboard.

Note: you can use the CXL calculator to determine if your experiment has reached significance.

Rollout plan

• Runtime: 30 days, or until statistical significance is reached
• Initial rollout: 50% of actors on GitLab.com (group integrations page only; projects page is a follow-up)

/chatops run feature set ordered_integrations 50 --actors

Status

Preferred workflow

1. PM determines and manages the status of the experiment (assign this issue to the PM)
2. PM asks for initial rollout on production, or changes to the status (assign to an Eng)
3. Eng changes the status using chatops (reassign to the PM)
4. When concluded, PM updates the Roll Out Steps and adds a milestone (assigns to an Eng)

Current status and history:

• API (GitLab team members)
• Feature flag log (GitLab team members)
• Experiment rollout board

Experiment Results

Roll Out Steps

• Confirm that end-to-end tests pass with the feature flag enabled.
  • See #e2e-run-staging Slack channel and look for the following messages:
    • test kicked off: Feature flag ordered_integrations has been set to true on **gstg**
    • test result: This pipeline was triggered due to toggling of ordered_integrations feature flag
• Enable on staging (/chatops run feature set ordered_integrations true --staging)
• Test on staging
• Ensure that documentation has been updated
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab ordered_integrations true)
• Coordinate a time to enable the flag with the SRE oncall and release managers
  • In #production mention @sre-oncall and @release-managers. Once confirmed, proceed with rollout
• Announce on the issue an estimated time this will be enabled on GitLab.com
• Enable on GitLab.com at 50% (/chatops run feature set ordered_integrations 50 --actors)
• Cross post chatops Slack command to #support_gitlab-com and #g_acquisition
• Announce on the issue that the flag has been enabled
• Remove experiment code and feature flag and add changelog entry - a separate cleanup issue might be required
• After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel
• Assign to the product manager to update the knowledge base (if applicable)

Rollback Steps

• This feature can be disabled by running the following Chatops command:

/chatops run feature set ordered_integrations false

Experiment Successful Cleanup Concerns

Items to be considered if candidate experience is to become a permanent part of GitLab

• Consider applying popularity ordering to the projects integrations page as a follow-up (tracked separately under the same experiment)