Enable cleanup policies for projects earlier than 12.7
Production Change
Change Summary
When we deployed cleanup policies, we excluded all the existing projects (eg. only new projects could have them). This was to slowly build the background jobs that will handle the load of executing them.
Then, we slowly included existing projects using a %
based feature flag. We reached 100%
a few weeks ago.
For self managed users, we let them choose if they wanted to let existing policies or not have cleanup policies. We did so by using an application setting. This setting was of course disabled
for GitLab.com (because we wanted to control the amount of cleanup policies created for existing projects through the feature flag)
This results in a function that checks if a given project (with no existing cleanup policy) can have one or not:
- Either the application setting is enabled or
- The feature flag is enabled for that project
Now for %14.9, we're going to remove that feature flag. The above function will basically read only the application setting. Given that the setting is disabled for GitLab.com, if we deploy that removal, we will basically exclude all older projects from having a cleanup policy. That's not what we want.
This production change is for enabling the application setting so that we can remove the feature flag.
Please note that because the function is using an or
with a 100%
enabled feature flag, this change has basically no immediate impact.
Change Details
- Services Impacted - ServiceGitLab Rails
- Change Technician - DRI for the execution of this change
- Change Reviewer - DRI for the review of this change
- Time tracking - 5 minutes
- Downtime Component - None
Detailed steps for the change
The application setting is available in the admin section of GitLab. To make sure that we're all on the same page, here is the UI with the application setting:
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 2mins
-
Navigate to the admin area, container registry settings: https://gitlab.com/admin/application_settings/ci_cd#js-registry-settings
. -
Verify that the checkbox Enable container expiration and retention policies for projects created earlier than GitLab 12.7.
is not checked.
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 2mins
-
Navigate to the admin area, container registry settings: https://gitlab.com/admin/application_settings/ci_cd#js-registry-settings
. -
Check the checkbox Enable container expiration and retention policies for projects created earlier than GitLab 12.7.
. -
Click the button Save changes
. -
Verify that the header Application settings saved successfully
appeared.
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 1min
-
Reload or navigate to the admin area, container registry settings: https://gitlab.com/admin/application_settings/ci_cd#js-registry-settings
. -
Verify that the checkbox Enable container expiration and retention policies for projects created earlier than GitLab 12.7.
is checked.
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 2mins
-
Navigate to the admin area, container registry settings: https://gitlab.com/admin/application_settings/ci_cd#js-registry-settings
. -
Uncheck the checkbox Enable container expiration and retention policies for projects created earlier than GitLab 12.7.
. -
Click the button Save changes
. -
Verify that the header Application settings saved successfully
appeared.
Monitoring
Again, this change has no immediate impact
Key metrics to observe
-
Kibana cleanup policies dashboard
- The daily rate of tags deleted should not drop below 5000.
- The daily peak (work: total) should not drop below 7000.
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.