Feature flag rollout: `ci_recreate_downstream_pipeline`

Summary

This issue is to rollout the feature on production, that is currently behind the ci_recreate_downstream_pipeline feature flag.

Owners

• Team: grouppipeline authoring
• Most appropriate slack channel to reach out to: #g_pipeline_authoring
• Best individual to reach out to: @lauraX (backend) or @mgandres (frontend)

Rollout Steps

Rollout on non-production environments

• Ensure that the feature MRs have been deployed to non-production environments.
  • /chatops run auto_deploy status <merge-commit-of-your-feature>
• Enable the feature globally on non-production environments.
  • /chatops run feature set ci_recreate_downstream_pipeline true --dev
  • /chatops run feature set ci_recreate_downstream_pipeline true --staging
• Verify that the feature works as expected. Posting the QA result in this issue is preferable.

Specific rollout on production

• Ensure that the feature MRs have been deployed to both production and canary.
  • /chatops run auto_deploy status <merge-commit-of-your-feature>
• Depending on the type of actor you are using, pick one of these options:
  • If you're using project-actor, you must enable the feature on these entries:
    • /chatops run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss,gitlab-com/www-gitlab-com ci_recreate_downstream_pipeline true
• Verify that the feature works on the specific entries. Posting the QA result in this issue is preferable.
• [ ]

Preparation before global rollout

• [-] Check if the feature flag change needs to be accompanied with a change management issue. Cross link the issue here if it does.
• Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the @sre-oncall Slack alias.
• [-] Ensure that documentation has been updated (More info).
• Announce on the feature issue an estimated time this will be enabled on GitLab.com.
• Ensure that any breaking changes have been announced following the release post process to ensure GitLab customers are aware.
• Notify #support_gitlab-com and your team channel (more guidance when this is necessary in the dev docs).

Global rollout on production

For visibility, all /chatops commands that target production should be executed in the #production slack channel and cross-posted (with the command results) to the responsible team's slack channel (#g_TEAM_NAME).

• Incrementally roll out the feature.
  • If the feature flag in code has an actor, perform actor-based rollout.
    • /chatops run feature set ci_recreate_downstream_pipeline 25 --actors
    • /chatops run feature set ci_recreate_downstream_pipeline 50 --actors
    • /chatops run feature set ci_recreate_downstream_pipeline 75 --actors
    • /chatops run feature set ci_recreate_downstream_pipeline 100 --actors
    • Enable the feature globally on production environment.
      • /chatops run feature set ci_recreate_downstream_pipeline true
• Announce on the feature issue that the feature has been globally enabled.
• Wait for at least one day for the verification term.

Release the feature

You can either create a follow-up issue for Feature Flag Cleanup or use the checklist below in this same issue.

• Create a merge request to remove <feature-flag-name> feature flag. Ask for review and merge it.
  • Remove all references to the feature flag from the codebase.
  • Remove the YAML definitions for the feature from the repository.
  • Create a changelog entry.
• Ensure that the cleanup MR has been deployed to both production and canary. If the merge request was deployed before the code cutoff, the feature can be officially announced in a release blog post.
  • /chatops run auto_deploy status <merge-commit-of-cleanup-mr>
• Close the feature issue to indicate the feature will be released in the current milestone.
• Clean up the feature flag from all environments by running these chatops command in #production channel:
  • /chatops run feature delete ci_recreate_downstream_pipeline --dev
  • /chatops run feature delete ci_recreate_downstream_pipeline --staging
  • /chatops run feature delete ci_recreate_downstream_pipeline

Rollback Steps

• This feature can be disabled by running the following Chatops command:

/chatops run feature set ci_recreate_downstream_pipeline false