Fix spec failures on CI when two connections

From %15.9, single DB connection is deprecated. We recommend two database connections to main: database. (Note: two databases is not yet ready for production use)

Running CI with two connections, we encounter the following failures (based on https://gitlab.com/gitlab-org/gitlab/-/pipelines/776667189)

RSpec failures (129 failures)

rspec './spec/graphql/types/notes/note_type_spec.rb[1:1]' # Types::Notes::NoteType adds rel="nofollow" to external links
rspec './spec/graphql/types/notes/note_type_spec.rb[1:2]' # Types::Notes::NoteType adds rel="noreferrer" to external links
rspec './spec/graphql/types/notes/note_type_spec.rb[1:3]' # Types::Notes::NoteType adds rel="noopener" to external links
rspec './spec/graphql/types/work_items/notes_filter_type_enum_spec.rb[1:1]' # Types::WorkItems::NotesFilterTypeEnum adds rel="nofollow" to external links
rspec './spec/graphql/types/work_items/notes_filter_type_enum_spec.rb[1:2]' # Types::WorkItems::NotesFilterTypeEnum adds rel="noreferrer" to external links
rspec './spec/graphql/types/work_items/notes_filter_type_enum_spec.rb[1:3]' # Types::WorkItems::NotesFilterTypeEnum adds rel="noopener" to external links
rspec './spec/graphql/types/work_items/widgets/notes_type_spec.rb[1:1]' # Types::WorkItems::Widgets::NotesType adds rel="nofollow" to external links
rspec './spec/graphql/types/work_items/widgets/notes_type_spec.rb[1:2]' # Types::WorkItems::Widgets::NotesType adds rel="noreferrer" to external links
rspec './spec/graphql/types/work_items/widgets/notes_type_spec.rb[1:3]' # Types::WorkItems::Widgets::NotesType adds rel="noopener" to external links
rspec './spec/lib/banzai/filter/absolute_link_filter_spec.rb[1:1]' # Banzai::Filter::AbsoluteLinkFilter adds rel="nofollow" to external links
rspec './spec/lib/banzai/filter/absolute_link_filter_spec.rb[1:2]' # Banzai::Filter::AbsoluteLinkFilter adds rel="noreferrer" to external links
rspec './spec/lib/banzai/filter/absolute_link_filter_spec.rb[1:3]' # Banzai::Filter::AbsoluteLinkFilter adds rel="noopener" to external links
rspec './spec/lib/banzai/filter/normalize_source_filter_spec.rb[1:1]' # Banzai::Filter::NormalizeSourceFilter adds rel="nofollow" to external links
rspec './spec/lib/banzai/filter/normalize_source_filter_spec.rb[1:2]' # Banzai::Filter::NormalizeSourceFilter adds rel="noreferrer" to external links
rspec './spec/lib/banzai/filter/normalize_source_filter_spec.rb[1:3]' # Banzai::Filter::NormalizeSourceFilter adds rel="noopener" to external links
rspec './spec/lib/banzai/filter/suggestion_filter_spec.rb[1:1]' # Banzai::Filter::SuggestionFilter adds rel="nofollow" to external links
rspec './spec/lib/banzai/filter/suggestion_filter_spec.rb[1:2]' # Banzai::Filter::SuggestionFilter adds rel="noreferrer" to external links
rspec './spec/lib/banzai/filter/suggestion_filter_spec.rb[1:3]' # Banzai::Filter::SuggestionFilter adds rel="noopener" to external links
rspec './spec/lib/banzai/reference_parser/alert_parser_spec.rb[1:1]' # Banzai::ReferenceParser::AlertParser adds rel="nofollow" to external links
rspec './spec/lib/banzai/reference_parser/alert_parser_spec.rb[1:2]' # Banzai::ReferenceParser::AlertParser adds rel="noreferrer" to external links
rspec './spec/lib/banzai/reference_parser/alert_parser_spec.rb[1:3]' # Banzai::ReferenceParser::AlertParser adds rel="noopener" to external links
rspec './spec/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables_spec.rb[1:1:2:2:2:1]' # Gitlab::Database::MigrationHelpers::AutomaticLockWritesOnTables when executing create_table migrations when multiple databases for creating a gitlab_ci table behaves like locks writes on table errors on deleting
rspec './spec/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables_spec.rb[1:2:2:2:1]' # Gitlab::Database::MigrationHelpers::AutomaticLockWritesOnTables when renaming a table when a gitlab_ci table behaves like locks writes on table errors on deleting
rspec './spec/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables_spec.rb[1:3:2:2:2:1]' # Gitlab::Database::MigrationHelpers::AutomaticLockWritesOnTables when reversing drop_table migrations when multiple databases for re-creating a gitlab_ci table behaves like locks writes on table errors on deleting
rspec './spec/lib/gitlab/database/migration_helpers/restrict_gitlab_schema_spec.rb[1:2:18:1:1:1]' # Gitlab::Database::MigrationHelpers::RestrictGitlabSchema when executing migrations does raise exception about cross schema access when suppressing restriction to ensure for db_config_name=main while restrict_gitlab_migration=no_gitlab_schema does run migrate :up and :down
rspec './spec/lib/gitlab/database/migration_helpers/restrict_gitlab_schema_spec.rb[1:2:18:1:2:1]' # Gitlab::Database::MigrationHelpers::RestrictGitlabSchema when executing migrations does raise exception about cross schema access when suppressing restriction to ensure for db_config_name=main while restrict_gitlab_migration=gitlab_schema_gitlab_main does run migrate :up and :down
rspec './spec/lib/gitlab/database/migration_helpers/restrict_gitlab_schema_spec.rb[1:2:18:1:3:1]' # Gitlab::Database::MigrationHelpers::RestrictGitlabSchema when executing migrations does raise exception about cross schema access when suppressing restriction to ensure for db_config_name=main while restrict_gitlab_migration=gitlab_schema_gitlab_shared does run migrate :up and :down
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:1]' # Gitlab::Database::Migrations::Runner ci database creates the results dir when one does not exist
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:2:1:1]' # Gitlab::Database::Migrations::Runner ci database .up result directory uses the /up subdirectory
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:2:2:1]' # Gitlab::Database::Migrations::Runner ci database .up migrations to run is the list of pending migrations
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:2:3:1]' # Gitlab::Database::Migrations::Runner ci database .up running migrations runs the unapplied migrations in regular/post order, then version order
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:2:3:2]' # Gitlab::Database::Migrations::Runner ci database .up running migrations writes a metadata file with the current schema version and database name
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:2:3:3]' # Gitlab::Database::Migrations::Runner ci database .up running migrations runs the unapplied migrations on the correct database
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:3:1:1]' # Gitlab::Database::Migrations::Runner ci database .down result directory is the /down subdirectory
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:3:2:1]' # Gitlab::Database::Migrations::Runner ci database .down migrations to run is the list of migrations that are up and on this branch
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:3:3:1]' # Gitlab::Database::Migrations::Runner ci database .down running migrations runs the applied migrations for the current branch in reverse order
rspec './spec/lib/gitlab/database/migrations/runner_spec.rb[1:3:3:4]' # Gitlab::Database::Migrations::Runner ci database .down writes a metadata file with the current schema version
rspec './spec/lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection_spec.rb[1:1:2:1]' # Gitlab::Database::QueryAnalyzers::GitlabSchemasValidateConnection properly observes all queries for query accessing gitlab_ci and gitlab_main is expected to raise /The query tried to access \["projects", "ci_builds"\]/
rspec './spec/lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection_spec.rb[1:1:3:1]' # Gitlab::Database::QueryAnalyzers::GitlabSchemasValidateConnection properly observes all queries for query accessing gitlab_ci and gitlab_main the gitlab_schemas is always ordered is expected to raise /The query tried to access \["ci_builds", "projects"\]/
rspec './spec/lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection_spec.rb[1:1:6:1]' # Gitlab::Database::QueryAnalyzers::GitlabSchemasValidateConnection properly observes all queries for query accessing CI table from main database is expected to raise /The query tried to access \["ci_builds"\]/
rspec './spec/lib/gitlab/database/tables_locker_spec.rb[1:2:1:1]' # Gitlab::Database::TablesLocker when running on multiple databases #lock_writes locks table in schema gitlab_ci and database main
rspec './spec/lib/gitlab/database/tables_locker_spec.rb[1:2:1:2]' # Gitlab::Database::TablesLocker when running on multiple databases #lock_writes locks table in schema gitlab_main and database ci
rspec './spec/lib/gitlab/database/tables_locker_spec.rb[1:2:1:3]' # Gitlab::Database::TablesLocker when running on multiple databases #lock_writes unlocks table in schema gitlab_main and database main
rspec './spec/lib/gitlab/database/tables_locker_spec.rb[1:2:1:4]' # Gitlab::Database::TablesLocker when running on multiple databases #lock_writes unlocks table in schema gitlab_ci and database ci
rspec './spec/lib/gitlab/database/tables_locker_spec.rb[1:2:1:5]' # Gitlab::Database::TablesLocker when running on multiple databases #lock_writes unlocks table in schema gitlab_shared and database main
rspec './spec/lib/gitlab/database/tables_locker_spec.rb[1:2:1:6]' # Gitlab::Database::TablesLocker when running on multiple databases #lock_writes unlocks table in schema gitlab_shared and database ci
rspec './spec/lib/gitlab/database/tables_locker_spec.rb[1:2:1:7]' # Gitlab::Database::TablesLocker when running on multiple databases #lock_writes unlocks table in schema gitlab_internal and database main
rspec './spec/lib/gitlab/database/tables_locker_spec.rb[1:2:1:8]' # Gitlab::Database::TablesLocker when running on multiple databases #lock_writes unlocks table in schema gitlab_internal and database ci
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:1:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are not locked for writes raises an error that the tables are not locked for writes
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:2:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are locked for writes truncates the legacy tables
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:2:2]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are locked for writes does not affect the other tables
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:2:3]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are locked for writes logs the sql statements to the logger
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:2:4:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are locked for writes when running in dry_run mode does not truncate the legacy tables if running in dry run mode
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:2:5:1:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are locked for writes when passing until_table parameter with a table that exists only truncates until the table specified
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:2:5:2:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are locked for writes when passing until_table parameter with a table that does not exist raises an error if the specified table does not exist
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:2:6:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are locked for writes when one of the attached partitions happened to be locked for writes truncates the locked partition successfully
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:1:1:2:7:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_ci tables on the main database behaves like truncating legacy tables on a database when the truncated tables are locked for writes with geo configured does not truncate gitlab_geo tables
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:1:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are not locked for writes raises an error that the tables are not locked for writes
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:2:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are locked for writes truncates the legacy tables
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:2:2]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are locked for writes does not affect the other tables
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:2:3]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are locked for writes logs the sql statements to the logger
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:2:4:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are locked for writes when running in dry_run mode does not truncate the legacy tables if running in dry run mode
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:2:5:1:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are locked for writes when passing until_table parameter with a table that exists only truncates until the table specified
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:2:5:2:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are locked for writes when passing until_table parameter with a table that does not exist raises an error if the specified table does not exist
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:2:6:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are locked for writes when one of the attached partitions happened to be locked for writes truncates the locked partition successfully
rspec './spec/lib/gitlab/database/tables_truncate_spec.rb[1:2:1:2:7:1]' # Gitlab::Database::TablesTruncate when truncating gitlab_main tables on the ci database behaves like truncating legacy tables on a database when the truncated tables are locked for writes with geo configured does not truncate gitlab_geo tables
rspec './spec/models/integrations/pipelines_email_spec.rb[1:3:1:4:2:2:1:1]' # Integrations::PipelinesEmail#execute with recipients when the pipeline failed on a protected branch notifications are enabled only for protected branch behaves like sending email sends email
rspec './spec/models/integrations/pipelines_email_spec.rb[1:3:1:4:2:3:1:1]' # Integrations::PipelinesEmail#execute with recipients when the pipeline failed on a protected branch notifications are enabled only for default and protected branches  behaves like sending email sends email
rspec './spec/models/work_items/widgets/hierarchy_spec.rb[1:1]' # WorkItems::Widgets::Hierarchy adds rel="nofollow" to external links
rspec './spec/models/work_items/widgets/hierarchy_spec.rb[1:2]' # WorkItems::Widgets::Hierarchy adds rel="noreferrer" to external links
rspec './spec/models/work_items/widgets/hierarchy_spec.rb[1:3]' # WorkItems::Widgets::Hierarchy adds rel="noopener" to external links
rspec './spec/services/issues/export_csv_service_spec.rb[1:5]' # Issues::ExportCsvService adds rel="nofollow" to external links
rspec './spec/services/issues/export_csv_service_spec.rb[1:6]' # Issues::ExportCsvService adds rel="noreferrer" to external links
rspec './spec/services/issues/export_csv_service_spec.rb[1:7]' # Issues::ExportCsvService adds rel="noopener" to external links
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:1:1:1:1]' # gitlab:db:truncate_legacy_tables when truncating ci tables on the main database behaves like truncating legacy tables when tables are not locked for writes raises an error when trying to truncate the tables
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:1:1:2:1]' # gitlab:db:truncate_legacy_tables when truncating ci tables on the main database behaves like truncating legacy tables when tables are locked for writes calls TablesTruncate with the correct parameters and default minimum batch size
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:1:1:2:2]' # gitlab:db:truncate_legacy_tables when truncating ci tables on the main database behaves like truncating legacy tables when tables are locked for writes truncates the legacy table
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:1:1:2:3]' # gitlab:db:truncate_legacy_tables when truncating ci tables on the main database behaves like truncating legacy tables when tables are locked for writes does not truncate the table that belongs to the connection schema
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:1:1:2:4:1]' # gitlab:db:truncate_legacy_tables when truncating ci tables on the main database behaves like truncating legacy tables when tables are locked for writes when running in dry_run mode does not truncate any tables
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:1:1:2:4:2]' # gitlab:db:truncate_legacy_tables when truncating ci tables on the main database behaves like truncating legacy tables when tables are locked for writes when running in dry_run mode prints the truncation sql statement to the output
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:1:1:2:5:1]' # gitlab:db:truncate_legacy_tables when truncating ci tables on the main database behaves like truncating legacy tables when tables are locked for writes when passing until_table parameter via environment variable sends the table name to TablesTruncate
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:2:1:1:1]' # gitlab:db:truncate_legacy_tables when truncating main tables on the ci database behaves like truncating legacy tables when tables are not locked for writes raises an error when trying to truncate the tables
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:2:1:2:1]' # gitlab:db:truncate_legacy_tables when truncating main tables on the ci database behaves like truncating legacy tables when tables are locked for writes calls TablesTruncate with the correct parameters and default minimum batch size
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:2:1:2:2]' # gitlab:db:truncate_legacy_tables when truncating main tables on the ci database behaves like truncating legacy tables when tables are locked for writes truncates the legacy table
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:2:1:2:3]' # gitlab:db:truncate_legacy_tables when truncating main tables on the ci database behaves like truncating legacy tables when tables are locked for writes does not truncate the table that belongs to the connection schema
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:2:1:2:4:1]' # gitlab:db:truncate_legacy_tables when truncating main tables on the ci database behaves like truncating legacy tables when tables are locked for writes when running in dry_run mode does not truncate any tables
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:2:1:2:4:2]' # gitlab:db:truncate_legacy_tables when truncating main tables on the ci database behaves like truncating legacy tables when tables are locked for writes when running in dry_run mode prints the truncation sql statement to the output
rspec './spec/tasks/gitlab/db/truncate_legacy_tables_rake_spec.rb[1:2:1:2:5:1]' # gitlab:db:truncate_legacy_tables when truncating main tables on the ci database behaves like truncating legacy tables when tables are locked for writes when passing until_table parameter via environment variable sends the table name to TablesTruncate
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:5:2:2:3:1:1:1]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs #perform when the base model does exist when the feature flag is enabled when active migrations exist when parallel execution is disabled when the calculated timeout is less than the minimum allowed sets the lease timeout to the minimum value
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:5:2:2:3:1:2]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs #perform when the base model does exist when the feature flag is enabled when active migrations exist when parallel execution is disabled always cleans up the exclusive lease
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:5:2:2:3:1:3]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs #perform when the base model does exist when the feature flag is enabled when active migrations exist when parallel execution is disabled delegetes the execution to ExecutionWorker
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:5:2:2:3:2:1]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs #perform when the base model does exist when the feature flag is enabled when active migrations exist when parallel execution is enabled delegetes the execution to ExecutionWorker
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:1:1:1]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is disabled behaves like batched background migration execution marks the migration record as finished
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:1:1:2]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is disabled behaves like batched background migration execution creates job records for each processed batch
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:1:1:3]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is disabled behaves like batched background migration execution marks all job records as succeeded
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:1:1:4]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is disabled behaves like batched background migration execution updates matching records in the range
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:1:1:6:1]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is disabled behaves like batched background migration execution health status puts migration on hold when there is autovaccum activity on related tables
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:1:1:6:2]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is disabled behaves like batched background migration execution health status puts migration on hold when the pending WAL count is above the limit
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:1:2]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is disabled assigns proper feature category to the context and the worker
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:2:1:1]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is enabled behaves like batched background migration execution marks the migration record as finished
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:2:1:2]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is enabled behaves like batched background migration execution creates job records for each processed batch
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:2:1:3]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is enabled behaves like batched background migration execution marks all job records as succeeded
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:2:1:4]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is enabled behaves like batched background migration execution updates matching records in the range
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:2:1:6:1]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is enabled behaves like batched background migration execution health status puts migration on hold when there is autovaccum activity on related tables
rspec './spec/workers/database/batched_background_migration/ci_database_worker_spec.rb[1:1:6:2:1:6:2]' # Database::BatchedBackgroundMigration::CiDatabaseWorker behaves like it runs batched background migration jobs executing an entire migration when parallel execution is enabled behaves like batched background migration execution health status puts migration on hold when the pending WAL count is above the limit
rspec ./spec/lib/backup/database_spec.rb:165 # Backup::Database#restore when not forced warns the user and waits
rspec ./spec/lib/backup/database_spec.rb:185 # Backup::Database#restore with an empty .gz file returns successfully
rspec ./spec/lib/backup/database_spec.rb:205 # Backup::Database#restore with a corrupted .gz file raises a backup error
rspec ./spec/lib/backup/database_spec.rb:221 # Backup::Database#restore when the restore command prints errors filters out noise from errors and has a post restore warning
rspec ./spec/lib/backup/database_spec.rb:248 # Backup::Database#restore with PostgreSQL settings defined in the environment overrides default config values
rspec ./spec/lib/gitlab/database/partitioning_spec.rb:119 # Gitlab::Database::Partitioning.sync_partitions with multiple databases creates partitions in each database
rspec ./spec/lib/gitlab/database/partitioning_spec.rb:178 # Gitlab::Database::Partitioning.sync_partitions when only a specific database is requested manages partitions for models for the given database
rspec ./spec/lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection_spec.rb:89 # Gitlab::Database::QueryAnalyzers::GitlabSchemasValidateConnection when analyzer is enabled for tests throws an error when trying to access a table that belongs to the gitlab_ci schema from the main database
rspec ./spec/lib/gitlab/database/tables_locker_spec.rb:163 # Gitlab::Database::TablesLocker when running on multiple databases when running in dry_run mode passes dry_run flag to LockManger
rspec ./spec/tasks/dev_rake_spec.rb:127 # dev rake tasks terminate_all_connections when a database is not found continues to next connection
rspec ./spec/tasks/dev_rake_spec.rb:155 # dev rake tasks multiple databases with a valid database copy_db:ci creates the database from main
rspec ./spec/tasks/dev_rake_spec.rb:167 # dev rake tasks multiple databases with a valid database copy_db:ci when the database already exists prints out a warning
rspec ./spec/tasks/gitlab/background_migrations_rake_spec.rb:141 # gitlab:background_migrations namespace rake tasks status when multiple database feature is enabled with a single database outputs the status of background migrations
rspec ./spec/tasks/gitlab/background_migrations_rake_spec.rb:78 # gitlab:background_migrations namespace rake tasks finalize when multiple database feature is enabled without the proper arguments exits without finalizing the migration
rspec ./spec/tasks/gitlab/background_migrations_rake_spec.rb:89 # gitlab:background_migrations namespace rake tasks finalize when multiple database feature is enabled with the proper arguments finalizes the matching migration
rspec ./spec/tasks/gitlab/db/decomposition/rollback/bump_ci_sequences_rake_spec.rb:92 # gitlab:db:decomposition:rollback:bump_ci_sequences when multiple databases does not change ci sequences on the ci database
rspec ./spec/tasks/gitlab/db_rake_spec.rb:732 # gitlab:db namespace rake task execute_async_index_operations delegates ci task to Gitlab::Database::AsyncIndexes
rspec ./spec/tasks/gitlab/db_rake_spec.rb:738 # gitlab:db namespace rake task execute_async_index_operations delegates ci task to Gitlab::Database::AsyncIndexes with specified argument
rspec ./spec/tasks/gitlab/db_rake_spec.rb:744 # gitlab:db namespace rake task execute_async_index_operations delegates main task to Gitlab::Database::AsyncIndexes
rspec ./spec/tasks/gitlab/db_rake_spec.rb:750 # gitlab:db namespace rake task execute_async_index_operations delegates main task to Gitlab::Database::AsyncIndexes with specified argument
rspec ./spec/tasks/gitlab/db_rake_spec.rb:756 # gitlab:db namespace rake task execute_async_index_operations delegates all task to every database with higher default for dev
rspec ./spec/tasks/gitlab/db_rake_spec.rb:763 # gitlab:db namespace rake task execute_async_index_operations delegates all task to every database with lower default for prod
rspec ./spec/tasks/gitlab/db_rake_spec.rb:772 # gitlab:db namespace rake task execute_async_index_operations delegates all task to every database with specified argument
rspec ./spec/tasks/gitlab/db_rake_spec.rb:780 # gitlab:db namespace rake task execute_async_index_operations when feature is not enabled is a no-op

Failing CI job logs: 776667189.zip

Unique files:

grep -h -E "^rspec " tmp/pipelines/776667189/* | sort | uniq | cut -d ' ' -f 2 | cut -d '[' -f 1 | cut -d ':' -f 1 | sort | uniq | sed "s/'//"

Proposal

Fix all failures above
Also, we should look to running with two connections on CI, maybe on the nightly schedule.
- Run database, migration specs, and background migration specs on CI !112383 (merged)
- Run unit specs on CI
- Run integration specs on CI
- Run feature specs on CI
I guess we stop running pipeline:run-single-db as often ?

Thanks

Thanks to @ayufan for this wonderful script to quickly get all job traces: https://gitlab.com/-/snippets/2490092

Edited Mar 09, 2023 by Omar Qunsul