[Feature Flag] Enable ci_optimize_project_records_destruction

Summary

This issue is to rollout optimized destruction of project-related CI records on production, that is currently behind the ci_optimize_project_records_destruction feature flag.

We have added a deleted_count to the AppLogger that should be 0 should everything go well. This piece of code should be removed as part of the cleanup of this rollout issue: #342925 (closed)

Owners

• Team: grouppipeline execution
• Most appropriate slack channel to reach out to: #g_pipeline_execution
• Best individual to reach out to: @drew or @lauraX
• PM: @jreporter

Stakeholders

This feature is also a step toward removing the dependent: :destroy association from Project to Ci::Build

Testing Groups/Projects/Users

• gitlab-org/gitlab project
• gitlab-org/gitlab-foss project
• gitlab-com/www-gitlab-com project
• gitlab-org/gitlab-com groups

Rollout Steps

Rollout on non-production environments

• Ensure that the feature MRs have been deployed to non-production environments.
  • /chatops run auto_deploy status 2ceeb0dd9c26a374be00c7c1274ecef768e401b0 10/19:
• Enable the feature globally on non-production environments.
  • /chatops run feature set ci_optimize_project_records_destruction true --dev
  • /chatops run feature set ci_optimize_project_records_destruction true --staging
• Verify that the feature works as expected. Posting the QA result in this issue is preferable.

Preparation before production rollout

• 10/19 (AM | UTC +2): Ensure that the feature MRs have been deployed to both production and canary.
  • /chatops run auto_deploy status 2ceeb0dd9c26a374be00c7c1274ecef768e401b0
• 10/19 (AM | UTC +2): Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production.
• 10/19 (AM | UTC +2): Announce on the feature issue an estimated time this will be enabled on GitLab.com.

Global rollout on production

• 10/19 (PM | UTC +2): Confirm the feature flag is enabled on staging without incident
• Roll out the feature to targeted testing projects/groups first
  • 10/19 (PM | UTC +2): /chatops run feature set --project=gitlab-org/gitlab ci_optimize_project_records_destruction true
  • 10/19 (PM | UTC +2): /chatops run feature set --project=gitlab-org/gitlab-foss ci_optimize_project_records_destruction true
  • 10/20 (AM | UTC +2): /chatops run feature set --project=gitlab-com/www-gitlab-com ci_optimize_project_records_destruction true
  • 10/20 (AM | UTC +2): /chatops run feature set ci_optimize_project_records_destruction 25 --actors
  • 10/20 (PM | UTC +2): /chatops run feature set ci_optimize_project_records_destruction 50 --actors
  • 10/21 (AM | UTC +2): /chatops run feature set ci_optimize_project_records_destruction 75 --actors
  • 10/21 (PM | UTC +2): /chatops run feature set ci_optimize_project_records_destruction 100 --actors
• Verify the change has the desired outcome with the limited rollout before enabling the feature globally on production.
• 10/22 (AM | UTC +2): Enable the feature globally on production environment. /chatops run feature set ci_optimize_project_records_destruction true
• Announce on the feature issue that the feature has been globally enabled.
• Wait for at least one day for the verification term.

Release the feature

• Create a merge request to remove ci_optimize_project_records_destruction feature flag. Ask for review and merge it.
  • Remove all references to the feature flag from the codebase.
  • Remove the YAML definitions for the feature from the repository.
  • Create a changelog entry.
• Ensure that the cleanup MR has been deployed to both production and canary. If the merge request was deployed before the code cutoff, the feature can be officially announced in a release blog post.
  • /chatops run auto_deploy status <merge-commit-of-cleanup-mr>
• Close the feature issue to indicate the feature will be released in the current milestone.
• Clean up the feature flag from all environments by running these chatops command in #production channel:
  • /chatops run feature delete ci_optimize_project_records_destruction --dev
  • /chatops run feature delete ci_optimize_project_records_destruction --staging
  • /chatops run feature delete ci_optimize_project_records_destruction
• Close this rollout issue.

Rollback Steps

• This feature can be disabled by running the following Chatops command:

/chatops run feature set ci_optimize_project_records_destruction false

Observability

Kibana
Kibana - AppLogger