CrossDatabaseModificationAcrossUnsupportedTablesError in Ci::UpdateBuildQueueService
Read about CrossDatabaseModificationAcrossUnsupportedTablesError at https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions . This example was discovered by Sentry logging in production.
Root cause
We update a namespaces
record while we are inside an open transaction that writes to the ci_builds
and ci_namespace_monthly_usages
table. This is not allowed as namespaces
will live in a different database to the other 2 tables when we finish ci-decomposition . As such we need to move this update to namespaces
outside of the transaction somehow.
Possible solutions
- Move this
extra_shared_runners_minutes_limit
column to a different table thannamespaces
which lives in theci_
database so that we can do all of this in a single transaction - Move the code to an after transaction callback
Read more at https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions .
First error details
Tables involved: ci_builds, ci_namespace_monthly_usages, namespaces
Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification::CrossDatabaseModificationAcrossUnsupportedTablesError
Cross-database data modification of 'gitlab_ci, gitlab_main' were detected within a transaction modifying the 'ci_builds, ci_namespace_monthly_usages, namespaces' tables.Please refer to https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions for details on how to resolve this exception.
Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification::CrossDatabaseModificationAcrossUnsupportedTablesError: Cross-database data modification of 'gitlab_ci, gitlab_main' were detected within a transaction modifying the 'ci_builds, ci_namespace_monthly_usages, namespaces' tables.Please refer to https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions for details on how to resolve this exception.
from lib/gitlab/database/query_analyzers/prevent_cross_database_modification.rb:101:in `analyze'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:62:in `perform'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:530:in `around_save'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:515:in `save!'
from activerecord (6.1.4.4) lib/active_record/persistence.rb:639:in `block in update!'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
from activerecord (6.1.4.4) lib/active_record/persistence.rb:637:in `update!'
from ee/app/models/ci/minutes/limit.rb:57:in `recalculate_remaining_purchased_minutes!'
from ee/app/models/ci/minutes/namespace_monthly_usage.rb:34:in `block (2 levels) in find_or_create_current'
from activesupport (6.1.4.4) lib/active_support/core_ext/object/try.rb:12:in `try'
from ee/app/models/ci/minutes/namespace_monthly_usage.rb:33:in `block in find_or_create_current'
from ee/app/models/ci/minutes/namespace_monthly_usage.rb:32:in `tap'
from ee/app/models/ci/minutes/namespace_monthly_usage.rb:32:in `find_or_create_current'
from ee/app/models/ci/minutes/quota.rb:84:in `current_usage'
from ee/app/models/ci/minutes/quota.rb:43:in `block (2 levels) in total_minutes_used'
from ee/app/models/ci/minutes/quota.rb:101:in `conditional_value'
from ee/app/models/ci/minutes/quota.rb:42:in `block in total_minutes_used'
from lib/gitlab/utils/strong_memoize.rb:28:in `strong_memoize'
from ee/app/models/ci/minutes/quota.rb:41:in `total_minutes_used'
from ee/app/models/ci/minutes/quota.rb:27:in `minutes_used_up?'
from ee/app/models/ee/ci/pending_build.rb:26:in `minutes_exceeded?'
from ee/app/models/ee/ci/pending_build.rb:19:in `args_from_build'
from app/models/ci/pending_build.rb:26:in `upsert_from_build!'
from app/models/ci/build.rb:1084:in `create_queuing_entry!'
from app/services/ci/update_build_queue_service.rb:22:in `block in push'
from state_machines (0.5.0) lib/state_machines/transition.rb:171:in `block in within_transaction'
from state_machines (0.5.0) lib/state_machines/machine.rb:1868:in `block in within_transaction'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:557:in `block in transaction'
from app/models/concerns/cross_database_modification.rb:58:in `block in transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:209:in `transaction'
from lib/gitlab/database.rb:265:in `block in transaction'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `block in instrument'
from activesupport (6.1.4.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `instrument'
from lib/gitlab/database.rb:264:in `transaction'
from app/models/concerns/cross_database_modification.rb:49:in `transaction'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:556:in `transaction'
from state_machines (0.5.0) lib/state_machines/machine.rb:1868:in `within_transaction'
from state_machines (0.5.0) lib/state_machines/transition.rb:170:in `within_transaction'
from app/services/ci/update_build_queue_service.rb:21:in `push'
from app/models/ci/build.rb:289:in `block (2 levels) in <class:Build>'
from state_machines (0.5.0) lib/state_machines/eval_helpers.rb:79:in `evaluate_method'
from state_machines (0.5.0) lib/state_machines/callback.rb:192:in `block in run_methods'
from state_machines (0.5.0) lib/state_machines/callback.rb:191:in `each'
from state_machines (0.5.0) lib/state_machines/callback.rb:191:in `run_methods'
from state_machines (0.5.0) lib/state_machines/callback.rb:159:in `call'
from state_machines (0.5.0) lib/state_machines/transition.rb:394:in `block (2 levels) in after'
from state_machines (0.5.0) lib/state_machines/transition.rb:394:in `each'
from state_machines (0.5.0) lib/state_machines/transition.rb:394:in `block in after'
from state_machines (0.5.0) lib/state_machines/transition.rb:392:in `catch'
from state_machines (0.5.0) lib/state_machines/transition.rb:392:in `after'
from state_machines (0.5.0) lib/state_machines/transition.rb:195:in `run_callbacks'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:127:in `run_callbacks'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:212:in `run_callbacks'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:63:in `catch'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:63:in `block in perform'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:182:in `block in within_transaction'
from state_machines (0.5.0) lib/state_machines/transition.rb:171:in `block in within_transaction'
from state_machines (0.5.0) lib/state_machines/machine.rb:1868:in `block in within_transaction'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:557:in `block in transaction'
from app/models/concerns/cross_database_modification.rb:58:in `block in transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:209:in `transaction'
from lib/gitlab/database.rb:265:in `block in transaction'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `block in instrument'
from activesupport (6.1.4.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `instrument'
from lib/gitlab/database.rb:264:in `transaction'
from app/models/concerns/cross_database_modification.rb:49:in `transaction'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:556:in `transaction'
from state_machines (0.5.0) lib/state_machines/machine.rb:1868:in `within_transaction'
from state_machines (0.5.0) lib/state_machines/transition.rb:170:in `within_transaction'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:181:in `within_transaction'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:62:in `perform'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:530:in `around_save'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:511:in `save'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:154:in `block (2 levels) in run_actions'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:154:in `each'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:154:in `block in run_actions'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:148:in `run_actions'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:60:in `perform'
from state_machines (0.5.0) lib/state_machines/transition.rb:163:in `perform'
from state_machines (0.5.0) lib/state_machines/event.rb:155:in `fire'
from state_machines (0.5.0) lib/state_machines/event.rb:219:in `block in add_actions'
from state_machines (0.5.0) lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'
from app/services/ci/process_build_service.rb:28:in `enqueue'
from ee/app/services/ee/ci/process_build_service.rb:23:in `enqueue'
from app/services/ci/process_build_service.rb:23:in `process'
from ee/app/services/ee/ci/process_build_service.rb:14:in `process'
from app/services/ci/process_build_service.rb:7:in `execute'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:87:in `block in update_processable!'
from lib/gitlab/optimistic_locking.rb:15:in `block in retry_lock'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:209:in `transaction'
from lib/gitlab/database.rb:265:in `block in transaction'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `block in instrument'
from activesupport (6.1.4.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `instrument'
from lib/gitlab/database.rb:264:in `transaction'
from app/models/concerns/cross_database_modification.rb:64:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:290:in `transaction'
from lib/gitlab/optimistic_locking.rb:14:in `retry_lock'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:85:in `update_processable!'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:88:in `each'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:88:in `each'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:65:in `update_processables!'
from activesupport (6.1.4.4) lib/active_support/core_ext/array/grouping.rb:39:in `block in in_groups_of'
from activesupport (6.1.4.4) lib/active_support/core_ext/array/grouping.rb:39:in `each'
from activesupport (6.1.4.4) lib/active_support/core_ext/array/grouping.rb:39:in `each_slice'
from activesupport (6.1.4.4) lib/active_support/core_ext/array/grouping.rb:39:in `in_groups_of'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:51:in `update_stage!'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:88:in `each'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:88:in `each'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:45:in `update_stages!'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:35:in `process!'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:22:in `block in execute'
from app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:22:in `execute'
from app/services/ci/process_pipeline_service.rb:16:in `execute'
from app/services/ci/pipeline_creation/start_pipeline_service.rb:13:in `execute'
from ee/app/services/ee/ci/pipeline_creation/start_pipeline_service.rb:12:in `execute'
from app/workers/ci/initial_pipeline_process_worker.rb:22:in `block in perform'
from activesupport (6.1.4.4) lib/active_support/core_ext/object/try.rb:12:in `try'
from app/workers/ci/initial_pipeline_process_worker.rb:19:in `perform'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:196:in `execute_job'
Second error
Tables involved: ci_builds, ci_namespace_monthly_usages, namespaces, ci_pending_builds
https://sentry.gitlab.net/gitlab/gitlabcom/issues/3177750/?referrer=gitlab_plugin
2nd error since cross-modification is detected for all queries in the transaction
Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification::CrossDatabaseModificationAcrossUnsupportedTablesError: Cross-database data modification of 'gitlab_ci, gitlab_main' were detected within a transaction modifying the 'ci_builds, ci_namespace_monthly_usages, namespaces, ci_pending_builds' tables.Please refer to https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions for details on how to resolve this exception.
lib/gitlab/database/query_analyzers/prevent_cross_database_modification.rb:101:in `analyze'
raise CrossDatabaseModificationAcrossUnsupportedTablesError, message
lib/gitlab/database/query_analyzer.rb:63:in `block in process_sql'
analyzer.analyze(parsed)
lib/gitlab/database/query_analyzer.rb:60:in `each'
analyzers.each do |analyzer|
lib/gitlab/database/query_analyzer.rb:60:in `process_sql'
analyzers.each do |analyzer|
lib/gitlab/database/query_analyzer.rb:28:in `block (2 levels) in hook!'
process_sql(event.payload[:sql], event.payload[:connection])
...
(244 additional frame(s) were not displayed)
Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification::CrossDatabaseModificationAcrossUnsupportedTablesError: Cross-database data modification of 'gitlab_ci, gitlab_main' were detected within a transaction modifying the 'ci_builds, ci_namespace_monthly_usages, namespaces, ci_pending_builds' tables.Please refer to https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions for details on how to resolve this exception.
from lib/gitlab/database/query_analyzers/prevent_cross_database_modification.rb:101:in `analyze'
from lib/gitlab/database/query_analyzer.rb:63:in `block in process_sql'
from lib/gitlab/database/query_analyzer.rb:60:in `each'
from lib/gitlab/database/query_analyzer.rb:60:in `process_sql'
from lib/gitlab/database/query_analyzer.rb:28:in `block (2 levels) in hook!'
from lib/gitlab/database/query_analyzer.rb:122:in `with_ignored_recursive_calls'
from lib/gitlab/database/query_analyzer.rb:27:in `block in hook!'
from activesupport (6.1.4.4) lib/active_support/notifications/fanout.rb:220:in `finish'
from activesupport (6.1.4.4) lib/active_support/notifications/fanout.rb:63:in `block in finish'
from activesupport (6.1.4.4) lib/active_support/notifications/fanout.rb:63:in `each'
from activesupport (6.1.4.4) lib/active_support/notifications/fanout.rb:63:in `finish'
from activesupport (6.1.4.4) lib/active_support/notifications/instrumenter.rb:45:in `finish_with_state'
from activesupport (6.1.4.4) lib/active_support/notifications/instrumenter.rb:30:in `instrument'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
from marginalia (1.10.0) lib/marginalia.rb:95:in `execute_and_clear_with_marginalia'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/postgresql/database_statements.rb:53:in `exec_query'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:154:in `exec_insert_all'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `exec_insert_all'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:95:in `method_missing'
from activerecord (6.1.4.4) lib/active_record/insert_all.rb:35:in `execute'
from activerecord (6.1.4.4) lib/active_record/persistence.rb:243:in `upsert_all'
from activerecord (6.1.4.4) lib/active_record/persistence.rb:188:in `upsert'
from app/models/ci/pending_build.rb:30:in `upsert_from_build!'
from app/models/ci/build.rb:1084:in `create_queuing_entry!'
from app/services/ci/update_build_queue_service.rb:22:in `block in push'
from state_machines (0.5.0) lib/state_machines/transition.rb:171:in `block in within_transaction'
from state_machines (0.5.0) lib/state_machines/machine.rb:1868:in `block in within_transaction'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:557:in `block in transaction'
from app/models/concerns/cross_database_modification.rb:58:in `block in transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:209:in `transaction'
from lib/gitlab/database.rb:265:in `block in transaction'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `block in instrument'
from activesupport (6.1.4.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `instrument'
from lib/gitlab/database.rb:264:in `transaction'
from app/models/concerns/cross_database_modification.rb:49:in `transaction'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:556:in `transaction'
from state_machines (0.5.0) lib/state_machines/machine.rb:1868:in `within_transaction'
from state_machines (0.5.0) lib/state_machines/transition.rb:170:in `within_transaction'
from app/services/ci/update_build_queue_service.rb:21:in `push'
from app/models/ci/build.rb:289:in `block (2 levels) in <class:Build>'
from state_machines (0.5.0) lib/state_machines/eval_helpers.rb:79:in `evaluate_method'
from state_machines (0.5.0) lib/state_machines/callback.rb:192:in `block in run_methods'
from state_machines (0.5.0) lib/state_machines/callback.rb:191:in `each'
from state_machines (0.5.0) lib/state_machines/callback.rb:191:in `run_methods'
from state_machines (0.5.0) lib/state_machines/callback.rb:159:in `call'
from state_machines (0.5.0) lib/state_machines/transition.rb:394:in `block (2 levels) in after'
from state_machines (0.5.0) lib/state_machines/transition.rb:394:in `each'
from state_machines (0.5.0) lib/state_machines/transition.rb:394:in `block in after'
from state_machines (0.5.0) lib/state_machines/transition.rb:392:in `catch'
from state_machines (0.5.0) lib/state_machines/transition.rb:392:in `after'
from state_machines (0.5.0) lib/state_machines/transition.rb:195:in `run_callbacks'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:127:in `run_callbacks'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:212:in `run_callbacks'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:63:in `catch'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:63:in `block in perform'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:182:in `block in within_transaction'
from state_machines (0.5.0) lib/state_machines/transition.rb:171:in `block in within_transaction'
from state_machines (0.5.0) lib/state_machines/machine.rb:1868:in `block in within_transaction'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:557:in `block in transaction'
from app/models/concerns/cross_database_modification.rb:58:in `block in transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:209:in `transaction'
from lib/gitlab/database.rb:265:in `block in transaction'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `block in instrument'
from activesupport (6.1.4.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `instrument'
from lib/gitlab/database.rb:264:in `transaction'
from app/models/concerns/cross_database_modification.rb:49:in `transaction'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:556:in `transaction'
from state_machines (0.5.0) lib/state_machines/machine.rb:1868:in `within_transaction'
from state_machines (0.5.0) lib/state_machines/transition.rb:170:in `within_transaction'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:181:in `within_transaction'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:62:in `perform'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:530:in `around_save'
from state_machines-activerecord (0.8.0) lib/state_machines/integrations/active_record.rb:511:in `save'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:154:in `block (2 levels) in run_actions'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:154:in `each'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:154:in `block in run_actions'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:148:in `run_actions'
from state_machines (0.5.0) lib/state_machines/transition_collection.rb:60:in `perform'
from state_machines (0.5.0) lib/state_machines/transition.rb:163:in `perform'
from state_machines (0.5.0) lib/state_machines/event.rb:155:in `fire'
from state_machines (0.5.0) lib/state_machines/event.rb:219:in `block in add_actions'
from state_machines (0.5.0) lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'
from app/services/ci/process_build_service.rb:28:in `enqueue'
from ee/app/services/ee/ci/process_build_service.rb:23:in `enqueue'
from app/services/ci/process_build_service.rb:23:in `process'
from ee/app/services/ee/ci/process_build_service.rb:14:in `process'
from app/services/ci/process_build_service.rb:7:in `execute'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:87:in `block in update_processable!'
from lib/gitlab/optimistic_locking.rb:15:in `block in retry_lock'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:209:in `transaction'
from lib/gitlab/database.rb:265:in `block in transaction'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `block in instrument'
from activesupport (6.1.4.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
from activesupport (6.1.4.4) lib/active_support/notifications.rb:203:in `instrument'
from lib/gitlab/database.rb:264:in `transaction'
from app/models/concerns/cross_database_modification.rb:64:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:290:in `transaction'
from lib/gitlab/optimistic_locking.rb:14:in `retry_lock'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:85:in `update_processable!'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:88:in `each'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:88:in `each'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:65:in `update_processables!'
from activesupport (6.1.4.4) lib/active_support/core_ext/array/grouping.rb:39:in `block in in_groups_of'
from activesupport (6.1.4.4) lib/active_support/core_ext/array/grouping.rb:39:in `each'
from activesupport (6.1.4.4) lib/active_support/core_ext/array/grouping.rb:39:in `each_slice'
from activesupport (6.1.4.4) lib/active_support/core_ext/array/grouping.rb:39:in `in_groups_of'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:51:in `update_stage!'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:88:in `each'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:88:in `each'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:45:in `update_stages!'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:35:in `process!'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:22:in `block in execute'
from app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'
from app/services/ci/pipeline_processing/atomic_processing_service.rb:22:in `execute'
from app/services/ci/process_pipeline_service.rb:16:in `execute'
from app/services/ci/pipeline_creation/start_pipeline_service.rb:13:in `execute'
from ee/app/services/ee/ci/pipeline_creation/start_pipeline_service.rb:12:in `execute'
from app/workers/ci/initial_pipeline_process_worker.rb:22:in `block in perform'
from activesupport (6.1.4.4) lib/active_support/core_ext/object/try.rb:12:in `try'
from app/workers/ci/initial_pipeline_process_worker.rb:19:in `perform'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:196:in `execute_job'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:138:in `block in invoke'
from lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'
from lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:58:in `perform'
from lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'
from lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `block in call'
from lib/gitlab/application_context.rb:74:in `block in use'
from gitlab-labkit (0.21.3) lib/labkit/context.rb:39:in `with_context'
from lib/gitlab/application_context.rb:74:in `use'
from lib/gitlab/application_context.rb:27:in `with_context'
from lib/gitlab/sidekiq_middleware/worker_context/server.rb:15:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'
from lib/gitlab/database/query_analyzer.rb:42:in `within'
from lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'
from lib/gitlab/with_request_store.rb:17:in `enabling_request_store'
from lib/gitlab/with_request_store.rb:10:in `with_request_store'
from lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/server_metrics.rb:66:in `block in call'
from lib/gitlab/sidekiq_middleware/server_metrics.rb:89:in `block in instrument'
from lib/gitlab/metrics/background_transaction.rb:33:in `run'
from lib/gitlab/sidekiq_middleware/server_metrics.rb:89:in `instrument'
from lib/gitlab/sidekiq_middleware/server_metrics.rb:65:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from gitlab-labkit (0.21.3) lib/labkit/middleware/sidekiq/server.rb:21:in `block in call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:138:in `block in invoke'
from gitlab-labkit (0.21.3) lib/labkit/middleware/sidekiq/context/server.rb:15:in `block in call'
from gitlab-labkit (0.21.3) lib/labkit/context.rb:39:in `with_context'
from gitlab-labkit (0.21.3) lib/labkit/middleware/sidekiq/context/server.rb:14:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:143:in `invoke'
from gitlab-labkit (0.21.3) lib/labkit/middleware/sidekiq/server.rb:20:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'
from lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'
from lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from marginalia (1.10.0) lib/marginalia/sidekiq_instrumentation.rb:10:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from sentry-raven (3.1.2) lib/raven/integrations/sidekiq/cleanup_middleware.rb:7:in `call'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
from sidekiq (6.4.0) lib/sidekiq/middleware/chain.rb:143:in `invoke'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:163:in `block in process'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
from sidekiq (6.4.0) lib/sidekiq/job_retry.rb:114:in `local'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
from sidekiq (6.4.0) lib/sidekiq/rails.rb:14:in `block in call'
from activesupport (6.1.4.4) lib/active_support/execution_wrapper.rb:88:in `wrap'
from activesupport (6.1.4.4) lib/active_support/reloader.rb:72:in `block in wrap'
from activesupport (6.1.4.4) lib/active_support/execution_wrapper.rb:88:in `wrap'
from activesupport (6.1.4.4) lib/active_support/reloader.rb:71:in `wrap'
from sidekiq (6.4.0) lib/sidekiq/rails.rb:13:in `call'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:257:in `stats'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
from lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
from sidekiq (6.4.0) lib/sidekiq/job_retry.rb:81:in `global'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:124:in `block in dispatch'
from sidekiq (6.4.0) lib/sidekiq/logger.rb:11:in `with'
from sidekiq (6.4.0) lib/sidekiq/job_logger.rb:33:in `prepare'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:123:in `dispatch'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:162:in `process'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:78:in `process_one'
from sidekiq (6.4.0) lib/sidekiq/processor.rb:68:in `run'
from sidekiq (6.4.0) lib/sidekiq/util.rb:56:in `watchdog'
from sidekiq (6.4.0) lib/sidekiq/util.rb:65:in `block in safe_thread'