Disable the certificate_based_clusters FF globally and re-enable it selectively
Production Change
Change Summary
We want to:
- Disable a feature flag (FF) globally.
- Re-enable this FF selectively for a list of Namespaces that should be fetched via a script
Change Details
- Services Impacted - ServiceRedis ServicePostgres chatops
- Change Technician - DRI for the execution of this change
- Change Reviewer - DRI for the review of this change
- Time tracking - expected max 18 minutes
- Downtime Component - No downtime expected
Detailed steps for the change
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 13 minutes
-
Set label changein-progress /label ~change::in-progress
-
Disable the FF globally Feature.disable(:certificate_based_clusters)
-
Enabled the FF for the 1st group Namespace.where(id: Deployment .where("deployments.created_at > '2022-04-03 00:00:00'") .where.not(cluster_id: nil) .joins('INNER JOIN cluster_groups on cluster_groups.cluster_id = deployments.cluster_id') .joins('INNER JOIN projects on projects.id = deployments.project_id') .joins('INNER JOIN namespaces on namespaces.id = projects.namespace_id') .select('DISTINCT(namespaces.id)') ).each_batch(of: 1000) do |namespaces_batch| namespaces_batch.each do |namespace| Feature.enable(:certificate_based_clusters, namespace) end end
-
Enabled the FF for the 2nd group Namespace.where(id: Deployment .where("deployments.created_at > '2022-04-03 00:00:00'") .where.not(cluster_id: nil) .joins('INNER JOIN cluster_projects on cluster_projects.cluster_id = deployments.cluster_id') .joins('INNER JOIN projects on projects.id = deployments.project_id') .joins('INNER JOIN namespaces on namespaces.id = projects.namespace_id') .select('DISTINCT(namespaces.id)') ).each_batch(of: 1000) do |namespaces_batch| namespaces_batch.each do |namespace| Feature.enable(:certificate_based_clusters, namespace) end end
-
Enabled the FF for the 3rd group Group.where(id: Group .select('DISTINCT(namespaces.id)') .joins('INNER JOIN cluster_groups on namespaces.id = cluster_groups.group_id') .joins('INNER JOIN clusters ON clusters.id = cluster_groups.cluster_id and clusters.enabled = true') ) .each_batch(of: 1000) do |namespaces_batch| namespaces_batch.each do |namespace| Feature.enable(:certificate_based_clusters, namespace) end end
-
Set label changecomplete /label ~change::complete
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 5 minutes
-
Disable the FF for the first group # Disables the FF for everyone. This is safer than removing the FF right away, because it will run 1 transaction per record that it needs to delete. The `Feature.remove` will open 1 transaction but execute multiple deletes, which might timeout. Feature.disable(:certificate_based_clusters) Feature.remove(:certificate_based_clusters)
Another approach could be to first delete the records from the table directly from Postgresql:
DELETE FROM feature_gates WHERE feature_key = 'certificate_based_clusters';
Then run the removal on Rails:
Feature.remove(:certificate_based_clusters)
-
Set label changeaborted /label ~change::aborted
Monitoring
Key metrics to observe
We might see small spikes during the execution of the scripts, but after it, everything should keep regular levels pre-script.
-
Metric: redis Service Apdex, saturation, memory saturation
- Location: https://dashboards.gitlab.net/d/redis-main/redis-overview?orgId=1
- What changes to this metric should prompt a rollback: Any abnormality after executing the scripts
-
Metric: Postgresql primary and replica CPU usage, errors, Replication lag, Primary active connections, DB CPU percent
- Location: https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1
- What changes to this metric should prompt a rollback: Any abnormality after executing the scripts
-
Metric: Latency of the Rails Web GroupsController
- Location: https://dashboards.gitlab.net/d/web-rails-controller/web-rails-controller?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-controller=GroupsController&var-action=show&from=now-3h&to=now
- What changes to this metric should prompt a rollback: Any abnormality after executing the scripts. Although, here a very small increase in latency might be acceptable. (dozens of milliseconds). Hundreds of ms should probably be rolled back.
Change Reviewer checklist
-
Check if the following applies: - The scheduled day and time of execution of the change is appropriate.
- The change plan is technically accurate.
- The change plan includes estimated timing values based on previous testing.
- The change plan includes a viable rollback plan.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
- The change plan includes success measures for all steps/milestones during the execution.
- The change adequately minimizes risk within the environment/service.
- The performance implications of executing the change are well-understood and documented.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
- If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
- The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
Check if all items below are complete: - The change plan is technically accurate.
- This Change Issue is linked to the appropriate Issue and/or Epic
- Change has been tested in staging and results noted in a comment on this issue.
- A dry-run has been conducted and results noted in a comment on this issue.
- For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In #production channel, mention
@sre-oncall
and this issue and await their acknowledgement.) - Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention
@release-managers
and this issue and await their acknowledgment.) - There are currently no active incidents that are severity1 or severity2
- If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.