[Feature flag] Rollout of `ci_reduce_queries_when_ticking_runner_queue`
What
Related to #322618 (closed)
Remove the :ci_reduce_queries_when_ticking_runner_queue
feature flag ...
Owners
- Team: ~"group::continuous integration"
- Most appropriate slack channel to reach out to:
#g_ci
- Best individual to reach out to: @fabiopitino
Expectations
What are we expecting to happen?
BuildQueueWorker
should run less queries and also spend less time in db since we remove an expensive query.
What might happen if this goes wrong?
We might see more jobs being requested by shared runners but the other hand we are already iterating over allowed runners, so we should not see anything going wrong.
What can we monitor to detect problems with this?
Percentiles of db counts by the worker: https://log.gprd.gitlab.net/goto/0007cca911d974681c873e2618ac9bfa
Overall performance of the worker: https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-queue=pipeline_processing:build_queue&from=1614755580000&to=1614777239999
Beta groups/projects
If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.
-
gitlab-org/gitlab
project -
gitlab-org
/gitlab-com
groups - ...
Roll Out Steps
-
Enable on staging ( /chatops run feature set feature_name true --staging
) -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab feature_name true
) -
Coordinate a time to enable the flag with the SRE oncall and release managers - In
#production
mention@sre-oncall
and@release-managers
. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
- In
-
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
(/chatops run feature set feature_name true
) -
Cross post chatops Slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry. Ensure that the feature flag definition YAML file has been removed in the same MR that is removing the feature flag from the code -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set --project=gitlab-org/gitlab feature_name false