[Feature flag] Rollout delayed_perform_for_build_hooks_worker feature flag
Feature
This feature uses the :delayed_perform_for_build_hooks_worker
feature flag!
It allows us to schedule BuildHooksWorker
3s in the future, to give the replication process behind #326095 (closed) time to complete.
Owners
- Team: groupmemory
- Most appropriate slack channel to reach out to:
#g_memory
- Best individual to reach out to: @nmilojevic1
- PM: @changzhengliu
Stakeholders
The Rollout Plan
- Partial Rollout on GitLab.com with beta groups
- Rollout on GitLab.com for a certain period (How long)
- Percentage Rollout on GitLab.com
- Rollout Feature for everyone as soon as it's ready
Beta Groups/Projects:
-
gitlab-org/gitlab
project -
gitlab-org
/gitlab-com
groups - ...
Expectations
What are we expecting to happen?
The job will be scheduled 3s in the future. In case of this worker, nothing is expected to happen. It should just allow enough time for replication process to finish, when we enable #326095 (closed)
What might happen if this goes wrong?
We just need to roll back the feature flag.
What can we monitor to detect problems with this?
- error ratio on Sidekiq nodes
- Metric: Queue lenghts
- Metric: Sidekiq queue detail dashboard
Rollout Timeline
Rollout Steps
Preparation Phase
-
Enable on staging ( /chatops run feature set delayed_perform_for_build_hooks_worker true --staging
) -
Test on staging -
Ensure that documentation has been updated (More info) -
Announce on the issue an estimated time this will be enabled on GitLab.com -
Check if the feature flag change needs to be accompanied with a change management issue. Cross link the issue here if it does. -
Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the @sre-oncall
Slack alias.
Partial Rollout Phase
-
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab feature_name true
) -
Verify behaviour (See Beta Groups) and add details with screenshots as a comment on this issue -
If it is possible to perform an incremental rollout, this should be preferred. Proposed increments are: 10%
,50%
,100%
. Proposed minimum time between increments is 15 minutes.- When setting percentages, make sure that the feature works correctly between feature checks. See #327117 (closed) for more information
- For actor-based rollout:
/chatops run feature set delayed_perform_for_build_hooks_worker 10 --actors
- For time-based rollout:
/chatops run feature set delayed_perform_for_build_hooks_worker 10
Full Rollout Phase
-
Make the feature flag enabled by default i.e. Change default_enabled
totrue
-
Cross post chatops slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Create a cleanup issue using the "Feature Flag Removal" template
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set delayed_perform_for_build_hooks_worker false