Feature flag rollout: `async_mr_pipeline_creation`

Summary

This issue is to roll out the async pipeline creation for new merge requests on production, that is currently behind the async_mr_pipeline_creation feature flag.

Related MR: !217381 (merged)

Owners

  • Most appropriate Slack channel to reach out to: #g_pipeline_authoring (new)
  • Best individual to reach out to: @lauraXD

Expectations

What are we expecting to happen?

When enabled, pipeline creation for new merge requests will be performed asynchronously via MergeRequests::CreatePipelineWorker instead of synchronously in NewMergeRequestWorker. This should reduce the P90 latency of NewMergeRequestWorker from over 3 seconds to under 1 second (S3 criteria target).

What can go wrong and how would we detect it?

  • Pipeline creation could fail silently or be delayed, causing merge requests to appear without pipelines longer than expected
  • Monitor NewMergeRequestWorker latency metrics and pipeline creation success rates
  • Watch for increased errors in MergeRequests::CreatePipelineWorker
  • Check for user reports of missing pipelines on newly created merge requests

Rollout Steps

Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command.

Rollout on non-production environments

  • Verify the MR with the feature flag is merged to master and has been deployed to non-production environments with /chatops run auto_deploy status <merge-commit-of-your-feature>
  • Enable the feature globally on non-production environments with /chatops run feature set async_mr_pipeline_creation true --dev --pre --staging --staging-ref
  • Enable the feature on the gitlab project etc /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss,gitlab-com/www-gitlab-com async_mr_pipeline_creation true
  • Verify that the feature works as expected. The best environment to validate the feature in is staging-canary as this is the first environment deployed to. Make sure you are configured to use canary.
  • If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking deployments.

Preparation before global rollout

  • Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable.
  • Check if the feature flag change needs to be accompanied with a change management issue. Cross link the issue here if it does.
  • Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production.
  • Incrementally roll out the feature on production.
    • /chatops gitlab run feature set async_mr_pipeline_creation 25 --actors
    • /chatops gitlab run feature set async_mr_pipeline_creation 50 --actors
    • /chatops gitlab run feature set async_mr_pipeline_creation 75 --actors
    • /chatops gitlab run feature set async_mr_pipeline_creation 100 --actors
    • Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net.
  • After the feature has been 100% enabled, wait for at least one day before releasing the feature.

Release the feature

  • Create a merge request to remove the async_mr_pipeline_creation feature flag. The MR should include:
    • Remove all references to the feature flag from the codebase.
    • Remove the YAML definitions for the feature from the repository.
  • Ensure that the cleanup MR has been included in the release package.
  • Once the cleanup MR has been deployed to production, clean up the feature flag from all environments: /chatops run feature delete async_mr_pipeline_creation --dev --pre --staging --staging-ref --production
  • Close this rollout issue.

Rollback Steps

  • This feature can be disabled on production by running the following Chatops command:
/chatops run feature set async_mr_pipeline_creation false
  • Disable the feature flag on non-production environments:
/chatops run feature set async_mr_pipeline_creation false --dev --pre --staging --staging-ref
  • Delete feature flag from all environments:
/chatops run feature delete async_mr_pipeline_creation --dev --pre --staging --staging-ref --production
Edited by Laura Montemayor