[CI] PostgreSQL deadlock while running specs on stable environments
Summary
We're seeing a deadlock in PostgreSQL when running specs on production/stable environments:
- https://gitlab.com/gitlab-org/charts/gitlab/-/jobs/3982394450 (covered in #4402 (closed))
- https://gitlab.com/gitlab-org/charts/gitlab/-/jobs/3949638324 (covered in #4377 (closed))
The only workaround we've found at the moment is to delete the PostgreSQL PVC and underlying PV and retry. If you retry before that, you get an error that the migrations have not run - and if you try to manually run migrations, you get:
PG::DuplicateSchema: ERROR: schema "gitlab_partitions_dynamic" already exists
We'll need to look into spec/features/backups_spec.rb
, as it looks like Sidekiq is trying to insert vulnerability_occurrences
while backup/restore is trying to drop/recreate related tables.
Logs
Logs for PIDs related to deadlock
[cluster: gke-cloud-native-v125]
~
$ k logs gke125-production-bz1tbp-postgresql-0 | grep 428417
Defaulted container "gke125-production-bz1tbp-postgresql" out of: gke125-production-bz1tbp-postgresql, metrics
2023-03-22 15:57:57.296 GMT [428417] ERROR: deadlock detected
2023-03-22 15:57:57.296 GMT [428417] DETAIL: Process 428417 waits for AccessExclusiveLock on relation 242544 of database 16384; blocked by process 427454.
Process 427454 waits for RowShareLock on relation 239129 of database 16384; blocked by process 428417.
Process 428417: /*application:web,db_config_name:main*/ DROP TABLE IF EXISTS "cluster_agent_tokens" CASCADE
2023-03-22 15:57:57.296 GMT [428417] HINT: See server log for query details.
2023-03-22 15:57:57.296 GMT [428417] STATEMENT: /*application:web,db_config_name:main*/ DROP TABLE IF EXISTS "cluster_agent_tokens" CASCADE
[cluster: gke-cloud-native-v125]
~
$ k logs gke125-production-bz1tbp-postgresql-0 | grep 427454
Defaulted container "gke125-production-bz1tbp-postgresql" out of: gke125-production-bz1tbp-postgresql, metrics
2023-03-22 15:57:23.956 GMT [427454] ERROR: duplicate key value violates unique constraint "index_events_on_target_type_and_target_id_and_fingerprint"
2023-03-22 15:57:23.956 GMT [427454] DETAIL: Key (target_type, target_id, fingerprint)=(WikiPage::Meta, 3, \x1b552eeb77ee9864464060e29924f48d4f9fb756) already exists.
2023-03-22 15:57:23.956 GMT [427454] STATEMENT: /*application:web,correlation_id:01GW50M2MWEKEG8X04H7ST69C0,endpoint_id:Projects::WikisController#create,db_config_name:main*/ INSERT INTO "events" ("project_id", "author_id", "target_id", "created_at", "updated_at", "action", "target_type", "fingerprint") VALUES (19, 1, 3, '2023-03-22 15:57:23.953475', '2023-03-22 15:57:23.953475', 1, 'WikiPage::Meta', '\x1b552eeb77ee9864464060e29924f48d4f9fb756') RETURNING "id"
2023-03-22 15:57:57.296 GMT [428417] DETAIL: Process 428417 waits for AccessExclusiveLock on relation 242544 of database 16384; blocked by process 427454.
Process 427454 waits for RowShareLock on relation 239129 of database 16384; blocked by process 428417.
Process 427454: /*application:web,correlation_id:01GW50N24KNHPWS05Y4JEJA2WC,endpoint_id:graphql:unknown,db_config_name:main*/ INSERT INTO "vulnerability_occurrences" ("created_at", "updated_at", "severity", "confidence", "report_type", "project_id", "scanner_id", "primary_identifier_id", "project_fingerprint", "location_fingerprint", "uuid", "name", "metadata_version", "raw_metadata", "vulnerability_id", "description") VALUES ('2023-03-22 15:57:56.378587', '2023-03-22 15:57:56.378587', 7, 2, 99, 27, 1, 1, '\x709584b2ef96c517fdffcbd5fdc200714b6de955', '\xc8e6b39154a0b9a9158fd949ba6ee0632aea8705', '256e8c9d-5be5-5e46-8972-308747c04fa6', 'Gryffindor vulnerability', 'manual:1.0', '{}', 1, 'Brave courageous and pompous vulnerability') RETURNING "id"
2023-03-22 15:57:57.841 GMT [427454] ERROR: relation "audit_events_external_audit_event_destinations" does not exist at character 152
2023-03-22 15:57:57.841 GMT [427454] STATEMENT: /*application:web,correlation_id:01GW50N4DSZT4MCP1WW48SPHGV,endpoint_id:POST /api/:version/internal/allowed,db_config_name:main*/ SELECT 1 AS one FROM "audit_events_external_audit_event_destinations" WHERE "audit_events_external_audit_event_destinations"."namespace_id" = 5 LIMIT 1
Logs around the time the deadlock happened
$ k logs gke125-production-bz1tbp-postgresql-0 | grep '2023-03-22 15:5[6-8]'
Defaulted container "gke125-production-bz1tbp-postgresql" out of: gke125-production-bz1tbp-postgresql, metrics
2023-03-22 15:57:23.956 GMT [427454] ERROR: duplicate key value violates unique constraint "index_events_on_target_type_and_target_id_and_fingerprint"
2023-03-22 15:57:23.956 GMT [427454] DETAIL: Key (target_type, target_id, fingerprint)=(WikiPage::Meta, 3, \x1b552eeb77ee9864464060e29924f48d4f9fb756) already exists.
2023-03-22 15:57:23.956 GMT [427454] STATEMENT: /*application:web,correlation_id:01GW50M2MWEKEG8X04H7ST69C0,endpoint_id:Projects::WikisController#create,db_config_name:main*/ INSERT INTO "events" ("project_id", "author_id", "target_id", "created_at", "updated_at", "action", "target_type", "fingerprint") VALUES (19, 1, 3, '2023-03-22 15:57:23.953475', '2023-03-22 15:57:23.953475', 1, 'WikiPage::Meta', '\x1b552eeb77ee9864464060e29924f48d4f9fb756') RETURNING "id"
2023-03-22 15:57:55.793 GMT [428157] ERROR: relation "appearances" does not exist at character 147
2023-03-22 15:57:55.793 GMT [428157] STATEMENT: /*application:web,correlation_id:01GW50N0T9M5A2G9F4EEH8DR1F,endpoint_id:ProjectsController#show,db_config_name:main*/ SELECT "appearances".* FROM "appearances" ORDER BY "appearances"."id" ASC LIMIT 1
2023-03-22 15:57:55.820 GMT [428157] ERROR: relation "appearances" does not exist at character 147
2023-03-22 15:57:55.820 GMT [428157] STATEMENT: /*application:web,correlation_id:01GW50N0T9M5A2G9F4EEH8DR1F,endpoint_id:ProjectsController#show,db_config_name:main*/ SELECT "appearances".* FROM "appearances" ORDER BY "appearances"."id" ASC LIMIT 1
2023-03-22 15:57:56.309 GMT [428220] ERROR: relation "application_setting_terms" does not exist at character 176
2023-03-22 15:57:56.309 GMT [428220] STATEMENT: /*application:web,correlation_id:01GW50N2X9ARJJX4AJ5MT2ZDZ8,endpoint_id:GET /api/:version/application/settings,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:57:57.128 GMT [428298] ERROR: relation "bulk_import_entities" does not exist at character 197
2023-03-22 15:57:57.128 GMT [428298] STATEMENT: /*application:sidekiq,correlation_id:7c5bc14931d3804f97dfba32bacbd11c,jid:dd44fb48af28af16c676e456,endpoint_id:BulkImports::EntityWorker,db_config_name:main*/ SELECT "bulk_import_entities".* FROM "bulk_import_entities" WHERE "bulk_import_entities"."id" = 2 LIMIT 1
2023-03-22 15:57:57.296 GMT [428417] ERROR: deadlock detected
2023-03-22 15:57:57.296 GMT [428417] DETAIL: Process 428417 waits for AccessExclusiveLock on relation 242544 of database 16384; blocked by process 427454.
Process 427454: /*application:web,correlation_id:01GW50N24KNHPWS05Y4JEJA2WC,endpoint_id:graphql:unknown,db_config_name:main*/ INSERT INTO "vulnerability_occurrences" ("created_at", "updated_at", "severity", "confidence", "report_type", "project_id", "scanner_id", "primary_identifier_id", "project_fingerprint", "location_fingerprint", "uuid", "name", "metadata_version", "raw_metadata", "vulnerability_id", "description") VALUES ('2023-03-22 15:57:56.378587', '2023-03-22 15:57:56.378587', 7, 2, 99, 27, 1, 1, '\x709584b2ef96c517fdffcbd5fdc200714b6de955', '\xc8e6b39154a0b9a9158fd949ba6ee0632aea8705', '256e8c9d-5be5-5e46-8972-308747c04fa6', 'Gryffindor vulnerability', 'manual:1.0', '{}', 1, 'Brave courageous and pompous vulnerability') RETURNING "id"
2023-03-22 15:57:57.296 GMT [428417] HINT: See server log for query details.
2023-03-22 15:57:57.296 GMT [428417] STATEMENT: /*application:web,db_config_name:main*/ DROP TABLE IF EXISTS "cluster_agent_tokens" CASCADE
2023-03-22 15:57:57.841 GMT [427454] ERROR: relation "audit_events_external_audit_event_destinations" does not exist at character 152
2023-03-22 15:57:57.841 GMT [427454] STATEMENT: /*application:web,correlation_id:01GW50N4DSZT4MCP1WW48SPHGV,endpoint_id:POST /api/:version/internal/allowed,db_config_name:main*/ SELECT 1 AS one FROM "audit_events_external_audit_event_destinations" WHERE "audit_events_external_audit_event_destinations"."namespace_id" = 5 LIMIT 1
2023-03-22 15:57:57.851 GMT [428204] ERROR: relation "authentication_events" does not exist at character 133
2023-03-22 15:57:57.851 GMT [428204] STATEMENT: /*application:web,correlation_id:01GW50N1MJ0K7QAHJJ50MXWA5P,endpoint_id:SessionsController#create,db_config_name:main*/ INSERT INTO "authentication_events" ("created_at", "user_id", "result", "ip_address", "provider", "user_name") VALUES ('2023-03-22 15:57:57.846456', 4, 1, '34.148.69.65/32', 'standard', 'QA User 4c284ddb14c0235a') RETURNING "id"
2023-03-22 15:57:58.495 GMT [424496] ERROR: relation "ci_namespace_mirrors" does not exist at character 176
2023-03-22 15:57:58.495 GMT [424496] STATEMENT: /*application:sidekiq,correlation_id:01GW50N532GB0MSRGWDTNKFAR4,jid:506096dba7f07545ab6ea30c,endpoint_id:Namespaces::ProcessSyncEventsWorker,db_config_name:main*/ INSERT INTO "ci_namespace_mirrors" ("namespace_id","traversal_ids") VALUES (63, '{63}') ON CONFLICT ("namespace_id") DO UPDATE SET "traversal_ids"=excluded."traversal_ids" RETURNING "id"
2023-03-22 15:57:58.774 GMT [428220] ERROR: relation "application_setting_terms" does not exist at character 176
2023-03-22 15:57:58.774 GMT [428220] STATEMENT: /*application:web,correlation_id:01GW50N5GSTFJYHJWZ99V4Q8K6,endpoint_id:GET /api/:version/application/settings,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:57:59.095 GMT [426957] ERROR: relation "award_emoji" does not exist at character 171
2023-03-22 15:57:59.095 GMT [426957] STATEMENT: /*application:sidekiq,correlation_id:01GW50N54HQVF2K2S35QZCC08R,jid:d31190303db7ddff6a9b5540,endpoint_id:NewIssueWorker,db_config_name:main*/ SELECT "award_emoji".* FROM "award_emoji" WHERE "award_emoji"."awardable_id" = 9 AND "award_emoji"."awardable_type" = 'Issue' ORDER BY "award_emoji"."id" ASC
2023-03-22 15:57:59.144 GMT [428204] ERROR: relation "appearances" does not exist at character 144
2023-03-22 15:57:59.144 GMT [428204] STATEMENT: /*application:web,correlation_id:01GW50N59BTZE6NKD4CZXS14TW,endpoint_id:RootController#index,db_config_name:main*/ SELECT "appearances".* FROM "appearances" ORDER BY "appearances"."id" ASC LIMIT 1
2023-03-22 15:57:59.175 GMT [428173] ERROR: relation "application_setting_terms" does not exist at character 176
2023-03-22 15:57:59.175 GMT [428173] STATEMENT: /*application:web,correlation_id:01GW50N5S5QYDTDEN44S4CJFZD,endpoint_id:PUT /api/:version/application/settings,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:57:59.178 GMT [428204] ERROR: relation "appearances" does not exist at character 144
2023-03-22 15:57:59.178 GMT [428204] STATEMENT: /*application:web,correlation_id:01GW50N59BTZE6NKD4CZXS14TW,endpoint_id:RootController#index,db_config_name:main*/ SELECT "appearances".* FROM "appearances" ORDER BY "appearances"."id" ASC LIMIT 1
2023-03-22 15:57:59.495 GMT [428141] ERROR: relation "award_emoji" does not exist at character 216
2023-03-22 15:57:59.495 GMT [428141] STATEMENT: /*application:web,correlation_id:01GW50N54HQVF2K2S35QZCC08R,endpoint_id:POST /api/:version/projects/:id/issues,db_config_name:main*/ SELECT "award_emoji"."name", "award_emoji"."awardable_id", COUNT(*) as count FROM "award_emoji" WHERE (name IN ('thumbsdown','thumbsup') AND awardable_type = 'Issue' AND awardable_id IN (9)) GROUP BY "award_emoji"."name", "award_emoji"."awardable_id"
2023-03-22 15:57:59.756 GMT [428220] ERROR: relation "application_setting_terms" does not exist at character 160
2023-03-22 15:57:59.756 GMT [428220] STATEMENT: /*application:web,correlation_id:01GW50N6FSZTYKZDDABE6HHSTG,endpoint_id:SessionsController#new,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:58:00.249 GMT [428380] ERROR: relation "batched_background_migrations" does not exist at character 454
2023-03-22 15:58:00.249 GMT [428380] STATEMENT: SELECT a.attname, format_type(a.atttypid, a.atttypmod),
2023-03-22 15:58:01.551 GMT [428141] ERROR: relation "application_setting_terms" does not exist at character 160
2023-03-22 15:58:01.551 GMT [428141] STATEMENT: /*application:web,correlation_id:01GW50N895165TWRPT04RP021C,endpoint_id:SessionsController#new,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:58:02.339 GMT [428220] ERROR: relation "application_setting_terms" does not exist at character 160
2023-03-22 15:58:02.339 GMT [428220] STATEMENT: /*application:web,correlation_id:01GW50N91W16GJK05WQ58JDQKK,endpoint_id:SessionsController#new,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:58:02.934 GMT [428252] ERROR: relation "bulk_imports" does not exist at character 180
2023-03-22 15:58:02.934 GMT [428252] STATEMENT: /*application:sidekiq,correlation_id:cba00c638dc83e5cc4b89492c2557475,jid:06f6c8e4659d39c4d45fe45d,endpoint_id:BulkImportWorker,db_config_name:main*/ SELECT "bulk_imports".* FROM "bulk_imports" WHERE "bulk_imports"."id" = 1 LIMIT 1
2023-03-22 15:58:03.217 GMT [428220] ERROR: relation "application_setting_terms" does not exist at character 160
2023-03-22 15:58:03.217 GMT [428220] STATEMENT: /*application:web,correlation_id:01GW50N9XBZ7MKW2Y2H3WXG02R,endpoint_id:SessionsController#new,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:58:03.332 GMT [422857] ERROR: relation "batched_background_migrations" does not exist at character 223
2023-03-22 15:58:03.332 GMT [422857] STATEMENT: /*application:sidekiq,correlation_id:36b6f7d456beae4fb0a16fb83aaa8759,jid:a25e6ec5f4e33c8afecfcba3,endpoint_id:Database::BatchedBackgroundMigrationWorker,db_config_name:main*/ SELECT "batched_background_migrations".* FROM "batched_background_migrations" WHERE "batched_background_migrations"."id" IN (SELECT DISTINCT ON (table_name) id FROM "batched_background_migrations" WHERE "batched_background_migrations"."gitlab_schema" IN ('gitlab_main', 'gitlab_ci', 'gitlab_shared', 'gitlab_internal', 'gitlab_pm', 'gitlab_main_clusterwide') AND ("batched_background_migrations"."status" IN (1)) AND (on_hold_until IS NULL OR on_hold_until < NOW()) ORDER BY "batched_background_migrations"."table_name" ASC, "batched_background_migrations"."id" ASC) ORDER BY "batched_background_migrations"."id" ASC LIMIT 1
2023-03-22 15:58:03.727 GMT [428220] ERROR: relation "application_setting_terms" does not exist at character 160
2023-03-22 15:58:03.727 GMT [428220] STATEMENT: /*application:web,correlation_id:01GW50NAD8QTQHCZ01W8DAE4VH,endpoint_id:SessionsController#new,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:58:03.911 GMT [428220] ERROR: relation "application_setting_terms" does not exist at character 160
2023-03-22 15:58:03.911 GMT [428220] STATEMENT: /*application:web,correlation_id:01GW50NAJZK8TQQEZY015DNV5Z,endpoint_id:SessionsController#new,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:58:09.299 GMT [428141] ERROR: relation "application_setting_terms" does not exist at character 160
2023-03-22 15:58:09.299 GMT [428141] STATEMENT: /*application:web,correlation_id:01GW50NFV2VNVEHR40NA8G91QK,endpoint_id:SessionsController#new,db_config_name:main*/ SELECT "application_setting_terms".* FROM "application_setting_terms" ORDER BY "application_setting_terms"."id" DESC LIMIT 1
2023-03-22 15:58:17.950 GMT [268753] ERROR: relation "ci_builds" does not exist at character 405
2023-03-22 15:58:17.950 GMT [268753] STATEMENT: SELECT
2023-03-22 15:58:17.953 GMT [268753] ERROR: relation "ci_builds" does not exist at character 405
2023-03-22 15:58:17.953 GMT [268753] STATEMENT: SELECT
2023-03-22 15:58:17.954 GMT [268753] ERROR: relation "ci_builds" does not exist at character 33
2023-03-22 15:58:17.954 GMT [268753] STATEMENT: SELECT
2023-03-22 15:58:17.955 GMT [268753] ERROR: relation "ci_builds" does not exist at character 521
2023-03-22 15:58:17.955 GMT [268753] STATEMENT: SELECT
2023-03-22 15:58:17.956 GMT [268753] ERROR: relation "ci_builds" does not exist at character 33
2023-03-22 15:58:17.956 GMT [268753] STATEMENT: SELECT
2023-03-22 15:58:18.592 GMT [422857] ERROR: relation "ci_namespace_mirrors" does not exist at character 182
2023-03-22 15:58:18.592 GMT [422857] STATEMENT: /*application:sidekiq,correlation_id:c9dbf0c5cdeb664f732be9358ac4ab38,jid:506096dba7f07545ab6ea30c,endpoint_id:Namespaces::ProcessSyncEventsWorker,db_config_name:main*/ INSERT INTO "ci_namespace_mirrors" ("namespace_id","traversal_ids") VALUES (63, '{63}') ON CONFLICT ("namespace_id") DO UPDATE SET "traversal_ids"=excluded."traversal_ids" RETURNING "id"
2023-03-22 15:58:18.663 GMT [426758] ERROR: relation "award_emoji" does not exist at character 177
2023-03-22 15:58:18.663 GMT [426758] STATEMENT: /*application:sidekiq,correlation_id:223fa13456456550afa67c61695c8083,jid:d31190303db7ddff6a9b5540,endpoint_id:NewIssueWorker,db_config_name:main*/ SELECT "award_emoji".* FROM "award_emoji" WHERE "award_emoji"."awardable_id" = 9 AND "award_emoji"."awardable_type" = 'Issue' ORDER BY "award_emoji"."id" ASC
2023-03-22 15:58:24.814 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.814 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'audit_events')) LIMIT 1
2023-03-22 15:58:24.818 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.818 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'web_hook_logs')) LIMIT 1
2023-03-22 15:58:24.820 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.820 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'loose_foreign_keys_deleted_records')) LIMIT 1
2023-03-22 15:58:24.823 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.823 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'batched_background_migration_job_transition_logs')) LIMIT 1
2023-03-22 15:58:24.825 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.825 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'p_ci_runner_machine_builds')) LIMIT 1
2023-03-22 15:58:24.828 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.828 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'incident_management_pending_alert_escalations')) LIMIT 1
2023-03-22 15:58:24.830 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.830 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'incident_management_pending_issue_escalations')) LIMIT 1
2023-03-22 15:58:24.832 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.832 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'security_findings')) LIMIT 1
2023-03-22 15:58:24.834 GMT [428465] ERROR: relation "postgres_partitioned_tables" does not exist at character 89
2023-03-22 15:58:24.834 GMT [428465] STATEMENT: /*application:sidekiq,db_config_name:main*/ SELECT "postgres_partitioned_tables".* FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), '.', 'verification_codes')) LIMIT 1
2023-03-22 15:58:30.697 GMT [424941] ERROR: relation "bulk_import_entities" does not exist at character 199
2023-03-22 15:58:30.697 GMT [424941] STATEMENT: /*application:sidekiq,correlation_id:71cb2a12dca9e3a699eb35699bbad63b,jid:c22acaca97c21f5f7c864884,endpoint_id:BulkImports::PipelineWorker,db_config_name:main*/ SELECT "bulk_import_entities".* FROM "bulk_import_entities" WHERE "bulk_import_entities"."id" = 2 LIMIT 1
2023-03-22 15:58:30.721 GMT [424941] ERROR: relation "bulk_import_entities" does not exist at character 197
2023-03-22 15:58:30.721 GMT [424941] STATEMENT: /*application:sidekiq,correlation_id:71cb2a12dca9e3a699eb35699bbad63b,jid:7abb16efc5344c52f9e128f3,endpoint_id:BulkImports::EntityWorker,db_config_name:main*/ SELECT "bulk_import_entities".* FROM "bulk_import_entities" WHERE "bulk_import_entities"."id" = 2 LIMIT 1
2023-03-22 15:58:40.164 GMT [423720] ERROR: relation "award_emoji" does not exist at character 177
2023-03-22 15:58:40.164 GMT [423720] STATEMENT: /*application:sidekiq,correlation_id:165dc78e8e36782d75c65e7960925654,jid:d31190303db7ddff6a9b5540,endpoint_id:NewIssueWorker,db_config_name:main*/ SELECT "award_emoji".* FROM "award_emoji" WHERE "award_emoji"."awardable_id" = 9 AND "award_emoji"."awardable_type" = 'Issue' ORDER BY "award_emoji"."id" ASC
2023-03-22 15:58:44.444 GMT [428251] ERROR: relation "ci_namespace_mirrors" does not exist at character 182
2023-03-22 15:58:44.444 GMT [428251] STATEMENT: /*application:sidekiq,correlation_id:d99d20b4c63293335b04696bd22c7fc6,jid:506096dba7f07545ab6ea30c,endpoint_id:Namespaces::ProcessSyncEventsWorker,db_config_name:main*/ INSERT INTO "ci_namespace_mirrors" ("namespace_id","traversal_ids") VALUES (63, '{63}') ON CONFLICT ("namespace_id") DO UPDATE SET "traversal_ids"=excluded."traversal_ids" RETURNING "id"
Edited by Mitchell Nielsen