fast_destroy_all ci_job_artifacts causes CrossDatabaseModificationAcrossUnsupportedTablesError with geo, and ci tables
Related to #346236 (closed).
See https://sentry.gitlab.net/gitlab/staginggitlabcom/issues/3167984/ :
Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification::CrossDatabaseModificationAcrossUnsupportedTablesError: Cross-database data modification of 'gitlab_ci, gitlab_main' were detected within a transaction modifying the 'ci_deleted_objects, ci_job_artifacts, geo_job_artifact_deleted_events' tables.Please refer to https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions for details on how to resolve this exception.
from ee/app/services/geo/job_artifact_deleted_event_store.rb:19:in `bulk_create'
from ee/app/services/ee/ci/job_artifacts/destroy_batch_service.rb:30:in `insert_geo_event_records'
from ee/app/services/ee/ci/job_artifacts/destroy_batch_service.rb:20:in `after_batch_destroy_hook'
from app/services/ci/job_artifacts/destroy_batch_service.rb:36:in `execute'
from app/services/ci/job_artifacts/destroy_associations_service.rb:16:in `block in destroy_records'
from app/models/concerns/each_batch.rb:98:in `block (2 levels) in each_batch'
from activerecord (6.1.4.4) lib/active_record/scoping/default.rb:34:in `block in unscoped'
from activerecord (6.1.4.4) lib/active_record/relation.rb:406:in `block in scoping'
from activerecord (6.1.4.4) lib/active_record/relation.rb:804:in `_scoping'
from activerecord (6.1.4.4) lib/active_record/relation.rb:406:in `scoping'
from activerecord (6.1.4.4) lib/active_record/scoping/default.rb:34:in `unscoped'
from app/models/concerns/each_batch.rb:98:in `block in each_batch'
from app/models/concerns/each_batch.rb:68:in `step'
from app/models/concerns/each_batch.rb:68:in `each_batch'
from app/services/ci/job_artifacts/destroy_associations_service.rb:14:in `destroy_records'
from app/models/ci/job_artifact.rb:281:in `begin_fast_destroy'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:67:in `block in begin_fast_destroy'
from activerecord (6.1.4.4) lib/active_record/relation.rb:406:in `block in scoping'
from activerecord (6.1.4.4) lib/active_record/relation.rb:804:in `_scoping'
from activerecord (6.1.4.4) lib/active_record/relation.rb:406:in `scoping'
from activerecord (6.1.4.4) lib/active_record/associations/collection_proxy.rb:1109:in `scoping'
from activerecord (6.1.4.4) lib/active_record/relation/delegation.rb:67:in `begin_fast_destroy'
from app/models/concerns/fast_destroy_all.rb:87:in `perform_fast_destroy'
from app/models/concerns/fast_destroy_all.rb:81:in `block in use_fast_destroy'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:427:in `instance_exec'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:427:in `block in make_lambda'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:603:in `catch'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:603:in `block in default_terminator'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:199:in `block in halting'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:512:in `block in invoke_before'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:512:in `each'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:512:in `invoke_before'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:105:in `run_callbacks'
from activesupport (6.1.4.4) lib/active_support/callbacks.rb:824:in `_run_destroy_callbacks'
from activerecord (6.1.4.4) lib/active_record/callbacks.rb:439:in `destroy'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:294:in `block in destroy'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
from activesupport (6.1.4.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
from activerecord (6.1.4.4) lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
from lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
from lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
from lib/gitlab/database/load_balancing/load_balancer.rb:179:in `retry_with_backoff'
from lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
from lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
from lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
from activerecord (6.1.4.4) lib/active_record/transactions.rb:294:in `destroy'
from activerecord (6.1.4.4) lib/active_record/persistence.rb:555:in `destroy!'
from app/services/ci/destroy_pipeline_service.rb:17:in `execute'
For now, I have added an allowlist in ee/app/services/ee/ci/job_artifacts/destroy_batch_service.rb
Proposal
See #346236 (comment 820315219) . This may also use a similar fix to !80096 (diffs)
Edited by Dylan Griffith