[FF] `pipeline_execution_schedule_policy_variables_override` - Rollout

Summary

This issue is to roll out the feature on production, that is currently behind the pipeline_execution_schedule_policy_variables_override feature flag.

Owners

  • Most appropriate Slack channel to reach out to: #g_security_policies
  • Best individual to reach out to: @Andyschoenen

Expectations

What are we expecting to happen?

When enabled, scheduled pipeline execution policy (PEP) jobs will include variables_override metadata that allows proper variable precedence handling. This aligns scheduled PEP behavior with regular PEP pipelines.

With variables_override: { allowed: true } (the default), project/group/instance CI/CD variables continue to take precedence over policy-defined variables, preserving existing behavior.

What can go wrong and how would we detect it?

  • Scheduled PEP pipelines could fail if variable handling changes unexpectedly
  • Monitor pipeline failure rates for scheduled PEP pipelines
  • Check Kibana logs for scheduled_pipeline_execution_policy_failure events

Rollout Steps

Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command.

Rollout on non-production environments

  • Verify the MR with the feature flag is merged to master and has been deployed to non-production environments with /chatops gitlab run auto_deploy status <merge-commit-of-your-feature>
  • Enable the feature globally on non-production environments with /chatops gitlab run feature set pipeline_execution_schedule_policy_variables_override true --dev --pre --staging --staging-ref
  • Verify that the feature works as expected.
  • If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking deployments.

Before production rollout

  • If the change is significant and you wanted to announce in #whats-happening-at-gitlab, it best to do it before rollout to gitlab-org/gitlab-com.

Specific rollout on production

  • Enable for gitlab-org group first: /chatops gitlab run feature set --group=gitlab-org pipeline_execution_schedule_policy_variables_override true
  • Verify that the feature works for the specific actors.

Preparation before global rollout

  • Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable.
  • Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production.

Global rollout on production

  • Incrementally roll out the feature on production.
    • /chatops gitlab run feature set pipeline_execution_schedule_policy_variables_override 25 --actors
    • Wait 15 minutes, monitor graphs
    • /chatops gitlab run feature set pipeline_execution_schedule_policy_variables_override 50 --actors
    • Wait 15 minutes, monitor graphs
    • /chatops gitlab run feature set pipeline_execution_schedule_policy_variables_override 100 --actors
  • After the feature has been 100% enabled, wait for at least one day before releasing the feature.

Release the feature

  • Create a merge request to remove the pipeline_execution_schedule_policy_variables_override feature flag.
  • Close the feature issue to indicate the feature will be released in the current milestone.
  • Clean up the feature flag from all environments: /chatops gitlab run feature delete pipeline_execution_schedule_policy_variables_override --dev --pre --staging --staging-ref --production
  • Close this rollout issue.

Rollback Steps

  • This feature can be disabled on production by running the following Chatops command:
/chatops gitlab run feature set pipeline_execution_schedule_policy_variables_override false