Remove Pending Deletion Projects from Production

Production Change

Change Summary

Incident #4790 (closed) is due to being unable to install the latest GitLab package due to projects that are still reporting themselves to be running without hashed storage configured. We have 254 projects that need to be cleaned up. This CR shall perform the removal of those projects

Change Details

1. Services Impacted - ServiceGitLab Rails
2. Change Technician - @skarbek
3. Change Reviewer - @igorwwwwwwwwwwwwwwwwwwww
4. Time tracking - Unknown - see below for details
5. Downtime Component - none

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 5

• Set label changein-progress on this issue
• Start a rails console session
• Validate the beginning state:

  Project.without_storage_feature(:repository).pluck(:id).count # should return 254
  project_ids = Project.without_storage_feature(:repository).pluck(:id)
  Project.where(id: project_ids, pending_delete: true).count # should return 254
  Project.where(id: project_ids, pending_delete: false).count # should return 0

These are projects that are okay to remove. So we'll proceed to do so below.

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - Unknown

⚠ Time is dependent on the procedure of removing the necessary project data from our systems. In staging this took approximately 20 seconds per project. We can assume at least 85 minutes, however, staging is fully of empty data and this may not be a good basis. ⚠

• In the existing rails console session

   admin_user = User.find_by_username('<your_admin_username>').id
   Project.where(id: project_ids, pending_delete: true).find_each do |i|
     ::Projects::DestroyService.new(i, admin_user, {}).execute
   end

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - Unknown

⚠ Time is dependent on the steps required for manual remediation ⚠

• Validate the end state:

  Project.without_storage_feature(:repository).pluck(:id).count # should return 0
  project_ids = Project.without_storage_feature(:repository).pluck(:id)
  Project.where(id: project_ids, pending_delete: true).count # should return 0

• If either value does not return 0, raise to #dev-escalation
• Begin manual remediation

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

⚠ there is no reversal of this procedure ⚠

These are projects that should have been removed, but for a variety of reasons they may not have on their own and are therefore dangling projects that are simply taking up storage space unnecessarily.

Monitoring

Key metrics to observe

n/a

Summary of infrastructure changes

• Does this change introduce new compute instances? No
• Does this change re-size any existing compute instances? No
• Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? No

Changes checklist

• This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
• This issue has the change technician as the assignee.
• Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
• Necessary approvals have been completed based on the Change Management Workflow.
• Change has been tested in staging and results noted in a comment on this issue.
• A dry-run has been conducted and results noted in a comment on this issue.
• SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
• There are currently no active incidents.