Enable container_registry_phase_2_deny_list feature flag in bulk

Production Change

Change Summary

We're about to start Phase 2 of the GitLab.com container registry upgrade/migration (gitlab-org&5523 (closed)).

As for Phase 1 (https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5770), we have a special feature flag that acts as deny list for VIP top-level namespaces that should be left to the end of the migration in order to reduce risk.

We need to enable this feature flag (container_registry_phase_2_deny_list) in bulk for all target top-level namespaces.

This is identical to https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5770, but here we use the ID instead of the path to identify the target top-level groups. This is done to account for potential group renames that may have occurred.

The list of target VIP top-level namespaces can be seen at https://gitlab.com/gitlab-com/gl-infra/production/-/snippets/2292214. This snippet contains a single JSON file with an array of integers, one for each top-level namespace ID.

This change is to execute a script on a production Rails console that will do the following:

Read JSON document from the private snippet.
Iterate over the list of top-level namespace IDs and for each:
1. Find the corresponding Group object by ID;
2. Enable container_registry_phase_2_deny_list feature flag for the Group.

This change won't trigger any application behavior, either on GitLab Rails or the Container Registry, as all the functionality sits behind the container_registry_migration_phase2 feature flag which is disabled. Additionally, we're starting Phase 2 by migrating repositories under the gitlab-org group first and then on the free tier only. We'll only move to paid tiers afterward.

Change Details

Services Impacted - ServiceGitLab Rails ServiceContainer Registry
Change Technician - @ahyield
Change Reviewer - @ahyield
Time tracking - 10 mins
Downtime Component - NA

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 5 minutes

Set label changein-progress on this issue
Obtain a PAT for the GitLab API with read access to private snippets in this project

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 5 minutes

Open rails console with write access in production

Execute the following script. Make sure to fill the PAT variable with your private access token and keep the console open for the validation and rollback steps if necessary:

PAT = "<TOKEN>"
SNIPPET_URL = "https://gitlab.com/api/v4/snippets/2292214/raw"
FEATURE_FLAG = :container_registry_phase_2_deny_list

response = Gitlab::HTTP.get(SNIPPET_URL, headers: { 'PRIVATE-TOKEN' => PAT })
ids = JSON.parse(response.body)

puts "Loaded #{ids.size} group IDs"

before_count = Feature::FlipperGate.where(feature_key: FEATURE_FLAG.to_s).count

ids.each do |id|
  group = Group.find_by_id(id)
  unless group
    puts "WARNING: #{id} not found!"
    next
  end
  Feature.enable(FEATURE_FLAG, group)
end

after_count = Feature::FlipperGate.where(feature_key: FEATURE_FLAG.to_s).count
diff_count = after_count - before_count

if diff_count == ids.size
  puts "enabled for {after_count} groups"
else
  puts "WARNING: Expected feature flag to be enabled for #{ids.size} groups, but only did for #{diff_count}"
end

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 1 Minute

No WARNING: .* messages should be seen during the execution of the script. Otherwise, report to @jdrpereira.
Run /chatops run feature get container_registry_phase_2_deny_list on Slack to ensure the feature flag scope was updated.

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 2 Minutes

Execute the following snippet:

before_count = Feature::FlipperGate.where(feature_key: FEATURE_FLAG.to_s).count

ids.each do |id|
    group = Group.find_by_id(id)
    Feature.disable(FEATURE_FLAG, group) if group
end

after_count = Feature::FlipperGate.where(feature_key: FEATURE_FLAG.to_s).count
unless after_count == 0
   puts "WARNING: Expected feature flag to be disabled for all groups, but #{after_count} remain"
end

Report any WARNING: .* messages to @jdrpereira.

Monitoring

Key metrics to observe

This change won't trigger any behavior change, either on GitLab Rails or the Container Registry, as all the functionality sits behind the container_registry_migration_phase2 feature flag. Additionally, we're starting Phase 2 by migrating repositories under the gitlab-org group first and then on the free tier only.

Change Reviewer checklist

C4 C3 C2 C1:

The scheduled day and time of execution of the change is appropriate.
The change plan is technically accurate.
The change plan includes estimated timing values based on previous testing.
The change plan includes a viable rollback plan.
The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
The change plan includes success measures for all steps/milestones during the execution.
The change adequately minimizes risk within the environment/service.
The performance implications of executing the change are well-understood and documented.
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
The change has a primary and secondary SRE with knowledge of the details available during the change window.

Change Technician checklist

Edited Apr 13, 2022 by Amy Phillips