Skip to content

Add worker to timeout status check responses

Jay Montal requested to merge 388725_timeout_pending_status_check_responses into master

What does this MR do and why?

What

Add cronjob worker to timeout status check responses

Why

Pending status check responses can become stale and we want to fail pending status check response after a 2 minute interval has passed so that users are informed sooner that their status checks have not passed

How to set up and validate locally

  1. Setup status checks
  2. Create an MR
  3. Manually run worker or wait for 2 minute timeout to occur

Migration runs

Adding `created_at` 
➜  gitlab git:(388725_timeout_pending_status_check_responses) ✗ rake db:migrate:up:main VERSION=20231005151816
WARNING: This version of GitLab depends on gitlab-shell 14.28.0, but you're running 14.27.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227020, pg_backend_pid: 79901
main: == 20231005151816 AddCreatedAtToStatusCheckResponses: migrating ===============
main: -- add_column(:status_check_responses, :created_at, :datetime_with_timezone, {:null=>false, :default=>#<Proc:0x0000000161185f58 /Users/jmontal/Workspaces/Work/gitlab-development-kit/gitlab/db/migrate/20231005151816_add_created_at_to_status_check_responses.rb:5 (lambda)>})
main:    -> 0.0018s
main: == 20231005151816 AddCreatedAtToStatusCheckResponses: migrated (0.0055s) ======

main: == [advisory_lock_connection] object_id: 227020, pg_backend_pid: 79901

➜  gitlab git:(388725_timeout_pending_status_check_responses) ✗ rake db:migrate:up:ci VERSION=20231005151816
WARNING: This version of GitLab depends on gitlab-shell 14.28.0, but you're running 14.27.0. Please update gitlab-shell.
ci: == [advisory_lock_connection] object_id: 227100, pg_backend_pid: 80366
ci: == 20231005151816 AddCreatedAtToStatusCheckResponses: migrating ===============
ci: -- add_column(:status_check_responses, :created_at, :datetime_with_timezone, {:null=>false, :default=>#<Proc:0x0000000164a86eb0 /Users/jmontal/Workspaces/Work/gitlab-development-kit/gitlab/db/migrate/20231005151816_add_created_at_to_status_check_responses.rb:5 (lambda)>})
ci:    -> 0.0022s
ci: == 20231005151816 AddCreatedAtToStatusCheckResponses: migrated (0.0133s) ======

ci: == [advisory_lock_connection] object_id: 227100, pg_backend_pid: 80366

➜  gitlab git:(388725_timeout_pending_status_check_responses) ✗ rake db:migrate:down:main VERSION=20231005151816
WARNING: This version of GitLab depends on gitlab-shell 14.28.0, but you're running 14.27.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227100, pg_backend_pid: 78991
main: == 20231005151816 AddCreatedAtToStatusCheckResponses: reverting ===============
main: -- remove_column(:status_check_responses, :created_at, :datetime_with_timezone, {:null=>false, :default=>#<Proc:0x000000015f4ff7f8 /Users/jmontal/Workspaces/Work/gitlab-development-kit/gitlab/db/migrate/20231005151816_add_created_at_to_status_check_responses.rb:5 (lambda)>})
main:    -> 0.0015s
main: == 20231005151816 AddCreatedAtToStatusCheckResponses: reverted (0.0069s) ======

main: == [advisory_lock_connection] object_id: 227100, pg_backend_pid: 78991

➜  gitlab git:(388725_timeout_pending_status_check_responses) ✗ rake db:migrate:down:ci VERSION=20231005151816
WARNING: This version of GitLab depends on gitlab-shell 14.28.0, but you're running 14.27.0. Please update gitlab-shell.
ci: == [advisory_lock_connection] object_id: 227080, pg_backend_pid: 79421
ci: == 20231005151816 AddCreatedAtToStatusCheckResponses: reverting ===============
ci: -- remove_column(:status_check_responses, :created_at, :datetime_with_timezone, {:null=>false, :default=>#<Proc:0x000000015c13ebd0 /Users/jmontal/Workspaces/Work/gitlab-development-kit/gitlab/db/migrate/20231005151816_add_created_at_to_status_check_responses.rb:5 (lambda)>})
ci:    -> 0.0019s
ci: == 20231005151816 AddCreatedAtToStatusCheckResponses: reverted (0.0156s) ======

ci: == [advisory_lock_connection] object_id: 227080, pg_backend_pid: 79421
Adding index on `id` & `status` 
➜  gitlab git:(388725_timeout_pending_status_check_responses) ✗ rake db:migrate:down:ci VERSION=20231017154804
WARNING: This version of GitLab depends on gitlab-shell 14.28.0, but you're running 14.27.0. Please update gitlab-shell.
ci: == [advisory_lock_connection] object_id: 226740, pg_backend_pid: 89703
ci: == 20231017154804 AddIndexToStatusCheckResponsesOnIdAndStatus: reverting ======
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.1007s
ci: -- indexes(:status_check_responses)
ci:    -> 0.0034s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0003s
ci: -- remove_index(:status_check_responses, {:algorithm=>:concurrently, :name=>"idx_status_check_responses_on_id_and_status"})
ci:    -> 0.0016s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0002s
ci: == 20231017154804 AddIndexToStatusCheckResponsesOnIdAndStatus: reverted (0.1270s)

ci: == [advisory_lock_connection] object_id: 226740, pg_backend_pid: 89703
➜  gitlab git:(388725_timeout_pending_status_check_responses) ✗ rake db:migrate:down:main VERSION=20231017154804
WARNING: This version of GitLab depends on gitlab-shell 14.28.0, but you're running 14.27.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 226740, pg_backend_pid: 90263
main: == 20231017154804 AddIndexToStatusCheckResponsesOnIdAndStatus: reverting ======
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0971s
main: -- indexes(:status_check_responses)
main:    -> 0.0032s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- remove_index(:status_check_responses, {:algorithm=>:concurrently, :name=>"idx_status_check_responses_on_id_and_status"})
main:    -> 0.0010s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20231017154804 AddIndexToStatusCheckResponsesOnIdAndStatus: reverted (0.1148s)

main: == [advisory_lock_connection] object_id: 226740, pg_backend_pid: 90263
➜  gitlab git:(388725_timeout_pending_status_check_responses) ✗ rake db:migrate:up:main VERSION=20231017154804
WARNING: This version of GitLab depends on gitlab-shell 14.28.0, but you're running 14.27.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 226680, pg_backend_pid: 90719
main: == 20231017154804 AddIndexToStatusCheckResponsesOnIdAndStatus: migrating ======
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0984s
main: -- index_exists?(:status_check_responses, [:id, :status], {:name=>"idx_status_check_responses_on_id_and_status", :algorithm=>:concurrently})
main:    -> 0.0033s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- add_index(:status_check_responses, [:id, :status], {:name=>"idx_status_check_responses_on_id_and_status", :algorithm=>:concurrently})
main:    -> 0.0015s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: == 20231017154804 AddIndexToStatusCheckResponsesOnIdAndStatus: migrated (0.1155s)

main: == [advisory_lock_connection] object_id: 226680, pg_backend_pid: 90719
➜  gitlab git:(388725_timeout_pending_status_check_responses) ✗ rake db:migrate:up:ci VERSION=20231017154804
WARNING: This version of GitLab depends on gitlab-shell 14.28.0, but you're running 14.27.0. Please update gitlab-shell.
ci: == [advisory_lock_connection] object_id: 226740, pg_backend_pid: 91190
ci: == 20231017154804 AddIndexToStatusCheckResponsesOnIdAndStatus: migrating ======
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.1030s
ci: -- index_exists?(:status_check_responses, [:id, :status], {:name=>"idx_status_check_responses_on_id_and_status", :algorithm=>:concurrently})
ci:    -> 0.0031s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0003s
ci: -- add_index(:status_check_responses, [:id, :status], {:name=>"idx_status_check_responses_on_id_and_status", :algorithm=>:concurrently})
ci:    -> 0.0022s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0003s
ci: == 20231017154804 AddIndexToStatusCheckResponsesOnIdAndStatus: migrated (0.1310s)

ci: == [advisory_lock_connection] object_id: 226740, pg_backend_pid: 91190

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #388725 (closed)

Edited by Jay Montal

Merge request reports