Archive old deployments (Delete old deployment refs)

What does this MR do and why?

GitLab Environment/Deployment support dynamic environment creation, that creates environments based on variables. One of the major usage of this feature is Review Apps. This type of environments are ephemeral that stopped after an associated merge request has been merged or after a certain period.

The performance implication on this feature is that deployment refs are going to be racked up and slows down git-fetch to the repository. So users have to periodically delete unnecessary environments to cascadingly delete deployment records/refs.

This performance issue can be surfaced even on a static environment. For example, production environment on gitlab-com/www-gitlab-com project has 554,097 deployment records, meaning the same number of refs exist. This environment would never be deleted, however, older deployments are highly unlikely never referenced.

We should automate and enforce the cleanup process at the system-level in order to assure the GitLab reliability.

This MR deletes old deployment refs. GitLab keeps the recent 50,000 of deployments and archive the rest.

Users still can fetch the deployed source code with git checkout <commit-sha> even after archive. Please see the documentation in this MR for more information.

Consider this is a similar feature with Archive Jobs.

Related #233882 (closed) #336746 (closed) #343454 (closed) #336926 (closed)

Sample query

Fetching the archivables:

SELECT
    "deployments".*
FROM
    "deployments"
WHERE
    "deployments"."project_id" = 53
    AND (
        iid <= ( 
            SELECT
                ("deployments"."iid" - 50000) AS start_iid
            FROM
                "deployments"
            WHERE
                "deployments"."project_id" = 53
            ORDER BY
                "deployments"."iid" DESC
            LIMIT
                1
        )
    )
)
AND "deployments"."archived" = FALSE
LIMIT
    100

Migration

shinya@shinya-Galago-Pro:~/workspace/thin-gdk/services/rails/src$ tre bin/rails db:migrate:redo VERSION=20211105010101
INFO: This script is a predefined script in devkitkat.
WARNING: This version of GitLab depends on gitlab-shell 13.21.1, but you're running 13.21.0. Please update gitlab-shell.
WARNING: This installation of GitLab uses a deprecated syntax for 'config/database.yml'. The support for this syntax will be removed in 15.0. More information can be found here: https://gitlab.com/gitlab-org/gitlab/-/issues/338182
== 20211105010101 AddArchivedColumnToDeployments: reverting ===================
-- remove_column(:deployments, :archived, :boolean, {:default=>false, :null=>false})
   -> 0.0028s
== 20211105010101 AddArchivedColumnToDeployments: reverted (0.0061s) ==========

== 20211105010101 AddArchivedColumnToDeployments: migrating ===================
-- add_column(:deployments, :archived, :boolean, {:default=>false, :null=>false})
   -> 0.0035s
== 20211105010101 AddArchivedColumnToDeployments: migrated (0.0035s) ==========

shinya@shinya-Galago-Pro:~/workspace/thin-gdk/services/rails/src$ tre bin/rails db:migrate:redo VERSION=20211110010101
INFO: This script is a predefined script in devkitkat.
WARNING: This version of GitLab depends on gitlab-shell 13.21.1, but you're running 13.21.0. Please update gitlab-shell.
WARNING: This installation of GitLab uses a deprecated syntax for 'config/database.yml'. The support for this syntax will be removed in 15.0. More information can be found here: https://gitlab.com/gitlab-org/gitlab/-/issues/338182
== 20211110010101 AddIndexOnUnarchivedDeployments: reverting ==================
-- transaction_open?()
   -> 0.0000s
-- indexes(:deployments)
   -> 0.0143s
-- remove_index(:deployments, {:algorithm=>:concurrently, :name=>"index_deployments_on_archived_project_id_iid"})
   -> 0.0087s
== 20211110010101 AddIndexOnUnarchivedDeployments: reverted (0.0258s) =========

== 20211110010101 AddIndexOnUnarchivedDeployments: migrating ==================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:deployments, [:archived, :project_id, :iid], {:name=>"index_deployments_on_archived_project_id_iid", :algorithm=>:concurrently})
   -> 0.0140s
-- add_index(:deployments, [:archived, :project_id, :iid], {:name=>"index_deployments_on_archived_project_id_iid", :algorithm=>:concurrently})
   -> 0.0123s
== 20211110010101 AddIndexOnUnarchivedDeployments: migrated (0.0284s) =========

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.