Skip to content

Use cronjob to delete old pages deployments

Kassio Borges requested to merge kassio/use-cronjob-to-delete-old-pages-deployments into master

What does this MR do and why?

Use cronjob to delete old pages deployments

Currently, the deletion of deprecated pages deployments is based on the idea that each project must have only 1 active PagesDeployment in a given moment. With the work to introduce Gitlab Pages Multiple Versions, a project now will have to support multiple PagesDeployment at the same time.

For this reason, PagesDeployment#deleted_at field was introduced to enable us to delete deprecated jobs by setting their deleted_at to the minimal timestamp when it's safe to delete them from the database.

Now, we're replacing the DestroyPagesDeploymentsWorker by Pages::DeprecatedDeploymentsDeleteCronWorker which will delete all depreacted PagesDeployment records, in other words, all PagesDeployment records that where deleted_at is not in the future.

Related to: #425939 (closed)

Changelog: changed

FAQ

  • Why deactive/deactivated instead of inactive, I wasn't sure, but I found this reference:
    • Inactivation Stopping action by changing an eviornment or changing form. Inactivation is most commonly used in relation to biological activities.
    • Deactivation The action of turning something off like a machine or tool. Generally speaking, deactivation is more permanent or long lasting than inactivation.
  • Why not remove the DestroyPagesDeploymentsWorker in this MR?
    • Two main reasons:
      1. To keep this MR smaller and more focused, I decided to remove DestroyPagesDeploymentsWorker on a follow-up
      2. There's some other old code that uses DestroyPagesDeploymentsWorker and that might be refactored when removing it. So, I think a follow up would be better for this purpose.

Database

Migration
bundle exec rails db:migrate:up:main VERSION=20230927141237 
main: == [advisory_lock_connection] object_id: 226900, pg_backend_pid: 50121
main: == 20230927141237 AddIndexOnPagesDeploymentsDeletedAt: migrating ==============
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0649s
main: -- index_exists?(:pages_deployments, :deleted_at, {:where=>"deleted_at IS NOT NULL", :name=>"index_pages_deployments_on_deleted_at", :algorithm=>:concurrently})
main:    -> 0.0116s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- add_index(:pages_deployments, :deleted_at, {:where=>"deleted_at IS NOT NULL", :name=>"index_pages_deployments_on_deleted_at", :algorithm=>:concurrently})
main:    -> 0.0081s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20230927141237 AddIndexOnPagesDeploymentsDeletedAt: migrated (0.1254s) =====

main: == [advisory_lock_connection] object_id: 226900, pg_backend_pid: 50121
bundle exec rails db:migrate:up:ci VERSION=20230927141237 
ci: == [advisory_lock_connection] object_id: 226960, pg_backend_pid: 50847
ci: == 20230927141237 AddIndexOnPagesDeploymentsDeletedAt: migrating ==============
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0628s
ci: -- index_exists?(:pages_deployments, :deleted_at, {:where=>"deleted_at IS NOT NULL", :name=>"index_pages_deployments_on_deleted_at", :algorithm=>:concurrently})
ci:    -> 0.0075s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0038s
ci: -- add_index(:pages_deployments, :deleted_at, {:where=>"deleted_at IS NOT NULL", :name=>"index_pages_deployments_on_deleted_at", :algorithm=>:concurrently})
ci:    -> 0.0078s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0002s
ci: == 20230927141237 AddIndexOnPagesDeploymentsDeletedAt: migrated (0.1250s) =====

ci: == [advisory_lock_connection] object_id: 226960, pg_backend_pid: 50847
bundle exec rails db:migrate:down:main VERSION=20230927141237 
main: == [advisory_lock_connection] object_id: 226960, pg_backend_pid: 70488
main: == 20230927141237 AddIndexOnPagesDeploymentsDeletedAt: reverting ==============
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0432s
main: -- index_exists?(:pages_deployments, :deleted_at, {:name=>"index_pages_deployments_on_deleted_at", :algorithm=>:concurrently})
main:    -> 0.0077s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0026s
main: -- remove_index(:pages_deployments, {:name=>"index_pages_deployments_on_deleted_at", :algorithm=>:concurrently, :column=>:deleted_at})
main:    -> 0.0145s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: == 20230927141237 AddIndexOnPagesDeploymentsDeletedAt: reverted (0.1087s) =====

main: == [advisory_lock_connection] object_id: 226960, pg_backend_pid: 70488
bundle exec rails db:migrate:down:ci VERSION=20230927141237 
ci: == [advisory_lock_connection] object_id: 226900, pg_backend_pid: 71175
ci: == 20230927141237 AddIndexOnPagesDeploymentsDeletedAt: reverting ==============
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.1164s
ci: -- index_exists?(:pages_deployments, :deleted_at, {:name=>"index_pages_deployments_on_deleted_at", :algorithm=>:concurrently})
ci:    -> 0.0101s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0022s
ci: -- remove_index(:pages_deployments, {:name=>"index_pages_deployments_on_deleted_at", :algorithm=>:concurrently, :column=>:deleted_at})
ci:    -> 0.0128s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0019s
ci: == 20230927141237 AddIndexOnPagesDeploymentsDeletedAt: reverted (0.2126s) =====

ci: == [advisory_lock_connection] object_id: 226900, pg_backend_pid: 71175

Screenshot

  • I changed the cron to run every minute, instead of 10 minutes
  • I changed deleted_at delay to be 30 seconds, instead of 30 minutes
  • The top right screen is query the database every second SELECT id, project_id, path_prefix, deleted_at FROM pages_deployments ORDER BY ID
  • The lower right screen tails the log/development to show the query (delete) that the cron job executes
  • I cut off the wait time from the video to make it shorter

full

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Kassio Borges

Merge request reports