Skip to content

Draft: fix: Wait and retry for current pod name during Geo failovers

Stephan Breitrainer requested to merge sb-fix-maintenancemode-in-geo into main

What does this MR do?

When doing a failover in Geo deployments, there is a race condition where updating the GitLab helm chart results in rotation of the toolbox pods. The pod name was picked up earlier and still holds the old pod name, which is already gone.
This was discovered during the work in Dedicated.

However, there is still one error to be addressed, which is hidden due to no_log: true and can be found in https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/instrumentor/-/jobs/5501712215#L1108:

/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': ERROR:  cannot execute UPDATE in a read-only transaction (PG::ReadOnlySqlTransaction)
See full error 
TASK [gitlab_geo : Maintenance Mode - Set Maintenance Mode for Cloud Native Hybrid environments] ***
fatal: [itestgeosandbox-consul-3 -> localhost]: FAILED! => changed=true 
  rc: 1
  return_code: 1
  stderr: |-
    /srv/gitlab/ee/app/services/remote_development/service_response_factory.rb:41: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/app/services/remote_development/workspaces/create_service.rb:28: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/app/services/remote_development/workspaces/reconcile_service.rb:38: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/app/services/remote_development/workspaces/update_service.rb:28: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/agent_config/main.rb:32: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/agent_config/updater.rb:11: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/authorizer.rb:12: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/authorizer.rb:13: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/creator.rb:17: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/creator.rb:24: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/creator.rb:25: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/creator.rb:28: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/devfile_fetcher.rb:15: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/devfile_fetcher.rb:16: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/devfile_flattener.rb:12: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/editor_component_injector.rb:12: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/editor_component_injector.rb:17: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/editor_component_injector.rb:18: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/post_flatten_devfile_validator.rb:43: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/post_flatten_devfile_validator.rb:54: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/post_flatten_devfile_validator.rb:102: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/post_flatten_devfile_validator.rb:126: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/post_flatten_devfile_validator.rb:159: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/post_flatten_devfile_validator.rb:202: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/post_flatten_devfile_validator.rb:235: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/pre_flatten_devfile_validator.rb:24: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/pre_flatten_devfile_validator.rb:52: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/project_cloner_component_injector.rb:12: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/project_cloner_component_injector.rb:17: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/project_cloner_component_injector.rb:18: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/project_cloner_component_injector.rb:23: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/volume_component_injector.rb:12: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/volume_component_injector.rb:13: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/create/volume_component_injector.rb:14: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/main.rb:43: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/update/authorizer.rb:12: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/update/updater.rb:12: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/input/agent_infos_observer.rb:15: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/input/params_extractor.rb:14: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/input/params_extractor.rb:16: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/input/params_to_infos_converter.rb:14: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/input/params_validator.rb:15: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/output/rails_infos_observer.rb:11: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/output/workspaces_to_rails_infos_converter.rb:15: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/persistence/orphaned_workspaces_observer.rb:12: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/persistence/workspaces_from_agent_infos_updater.rb:15: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/persistence/workspaces_to_be_returned_finder.rb:16: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/ee/lib/remote_development/workspaces/reconcile/persistence/workspaces_to_be_returned_updater.rb:11: warning: One-line pattern matching is experimental, and the behavior may change in future versions of Ruby!
    /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': PG::ReadOnlySqlTransaction: ERROR:  cannot execute UPDATE in a read-only transaction (ActiveRecord::StatementInvalid)
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:769:in `block in log'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:760:in `log'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/marginalia-1.11.1/lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:67:in `exec_delete'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:175:in `update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:513:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:1058:in `_update_row'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/locking/optimistic.rb:93:in `_update_row'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:1080:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/attribute_methods/dirty.rb:216:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/callbacks.rb:463:in `block in _update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/callbacks.rb:107:in `run_callbacks'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/callbacks.rb:929:in `_run_update_callbacks'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/callbacks.rb:463:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/timestamp.rb:121:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:1067:in `create_or_update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/callbacks.rb:455:in `block in create_or_update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/callbacks.rb:107:in `run_callbacks'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/callbacks.rb:455:in `create_or_update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/timestamp.rb:126:in `create_or_update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:648:in `save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/validations.rb:53:in `save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:302:in `block in save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:302:in `save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/suppressor.rb:54:in `save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:780:in `block in update!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:778:in `update!'
            from /srv/gitlab/lib/gitlab/current_settings.rb:32:in `method_missing'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/commands/runner/runner_command.rb:46:in `<main>'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/commands/runner/runner_command.rb:46:in `eval'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/commands/runner/runner_command.rb:46:in `perform'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/command/base.rb:87:in `perform'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/command.rb:48:in `invoke'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/commands.rb:18:in `<main>'
            from <internal:/usr/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
            from <internal:/usr/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
            from bin/rails:4:in `<main>'
    /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': ERROR:  cannot execute UPDATE in a read-only transaction (PG::ReadOnlySqlTransaction)
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:769:in `block in log'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract_adapter.rb:760:in `log'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/marginalia-1.11.1/lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:67:in `exec_delete'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:175:in `update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:513:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:1058:in `_update_row'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/locking/optimistic.rb:93:in `_update_row'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:1080:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/attribute_methods/dirty.rb:216:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/callbacks.rb:463:in `block in _update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/callbacks.rb:107:in `run_callbacks'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/callbacks.rb:929:in `_run_update_callbacks'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/callbacks.rb:463:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/timestamp.rb:121:in `_update_record'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:1067:in `create_or_update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/callbacks.rb:455:in `block in create_or_update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/callbacks.rb:107:in `run_callbacks'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/callbacks.rb:455:in `create_or_update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/timestamp.rb:126:in `create_or_update'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:648:in `save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/validations.rb:53:in `save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:302:in `block in save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:302:in `save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/suppressor.rb:54:in `save!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:780:in `block in update!'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
            from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
            from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.6/lib/active_record/persistence.rb:778:in `update!'
            from /srv/gitlab/lib/gitlab/current_settings.rb:32:in `method_missing'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/commands/runner/runner_command.rb:46:in `<main>'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/commands/runner/runner_command.rb:46:in `eval'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/commands/runner/runner_command.rb:46:in `perform'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/command/base.rb:87:in `perform'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/command.rb:48:in `invoke'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/railties-7.0.6/lib/rails/commands.rb:18:in `<main>'
            from <internal:/usr/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
            from <internal:/usr/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
            from /srv/gitlab/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
            from bin/rails:4:in `<main>'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
...ignoring

Related issues

Author's checklist

When ready for review, the Author applies the workflowready for review label and mention @gl-quality/get-maintainers:

  • Merge request:
    • Corresponding Issue raised and reviewed by the GET maintainers team.
    • Merge Request Title and Description are up-to-date, accurate, and descriptive
    • MR targeting the appropriate branch
    • MR has a green pipeline
    • MR has no new security alerts in the widget from the Secret Detection and IaC Scan (SAST) jobs.
  • Code:
    • Check the area changed works as expected. Consider testing it in different environment sizes (1k,3k,10k,etc.).
    • Documentation created/updated in the same MR.
    • If this MR adds an optional configuration - check that all permutations continue to work.
    • For Terraform changes: set up a previous version environment, then run a terraform plan with your new changes and ensure nothing will be destroyed. If anything will be destroyed and this can't be avoided please add a comment to the current MR.
  • Create any follow-up issue(s) to support the new feature across other supported cloud providers or advanced configurations. Create 1 issue for each provider/configuration. Contact the Quality Enablement team if unsure.

Merge request reports