Broken master: spec/migrations/schedule_fixing_security_scan_statuses_spec.rb
Summary
Master-broken incident: gitlab-org/quality/engineering-productivity/master-broken-incidents#1106 (closed)
Job #3907567768 failed for e4e27ed4:
Failures:
1) ScheduleFixingSecurityScanStatuses#up when the migration does not run on GitLab.com or development environment does not schedule the migration
Failure/Error: connection.public_send(...)
ActiveRecord::StatementInvalid:
PG::SREModifyingSqlDataNotPermitted: ERROR: Table: "ci_pipelines" is write protected within this Gitlab database.
HINT: Make sure you are using the right database connection
CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
# ./spec/migrations/schedule_fixing_security_scan_statuses_spec.rb:17:in `block (2 levels) in <main>'
# ./spec/spec_helper.rb:506:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:498:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:494:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:494:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:242:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/migration.rb:53:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:108:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:62:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:108:in `block (2 levels) in <main>'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:6:in `block in with_gitlab_schemas_validate_connection_prevented'
# ./lib/gitlab/database/query_analyzers/base.rb:39:in `with_suppressed'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:5:in `with_gitlab_schemas_validate_connection_prevented'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:11:in `block (2 levels) in <main>'
# ------------------
# --- Caused by: ---
# PG::SREModifyingSqlDataNotPermitted:
# ERROR: Table: "ci_pipelines" is write protected within this Gitlab database.
# HINT: Make sure you are using the right database connection
# CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
2) ScheduleFixingSecurityScanStatuses#up when the migration runs on GitLab.com behaves like scheduler for fixing the security scans status schedules background job
Failure/Error: connection.public_send(...)
ActiveRecord::StatementInvalid:
PG::SREModifyingSqlDataNotPermitted: ERROR: Table: "ci_pipelines" is write protected within this Gitlab database.
HINT: Make sure you are using the right database connection
CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE
Shared Example Group: "scheduler for fixing the security scans status" called from ./spec/migrations/schedule_fixing_security_scan_statuses_spec.rb:60
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
# ./spec/migrations/schedule_fixing_security_scan_statuses_spec.rb:17:in `block (2 levels) in <main>'
# ./spec/spec_helper.rb:506:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:498:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:494:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:494:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:242:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/migration.rb:53:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:108:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:62:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:108:in `block (2 levels) in <main>'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:6:in `block in with_gitlab_schemas_validate_connection_prevented'
# ./lib/gitlab/database/query_analyzers/base.rb:39:in `with_suppressed'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:5:in `with_gitlab_schemas_validate_connection_prevented'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:11:in `block (2 levels) in <main>'
# ------------------
# --- Caused by: ---
# PG::SREModifyingSqlDataNotPermitted:
# ERROR: Table: "ci_pipelines" is write protected within this Gitlab database.
# HINT: Make sure you are using the right database connection
# CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
3) ScheduleFixingSecurityScanStatuses#up when the migration runs on dev environment behaves like scheduler for fixing the security scans status schedules background job
Failure/Error: connection.public_send(...)
ActiveRecord::StatementInvalid:
PG::SREModifyingSqlDataNotPermitted: ERROR: Table: "ci_pipelines" is write protected within this Gitlab database.
HINT: Make sure you are using the right database connection
CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE
Shared Example Group: "scheduler for fixing the security scans status" called from ./spec/migrations/schedule_fixing_security_scan_statuses_spec.rb:66
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
# ./spec/migrations/schedule_fixing_security_scan_statuses_spec.rb:17:in `block (2 levels) in <main>'
# ./spec/spec_helper.rb:506:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:498:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:494:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:494:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:242:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/migration.rb:53:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:108:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:62:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:108:in `block (2 levels) in <main>'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:6:in `block in with_gitlab_schemas_validate_connection_prevented'
# ./lib/gitlab/database/query_analyzers/base.rb:39:in `with_suppressed'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:5:in `with_gitlab_schemas_validate_connection_prevented'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:11:in `block (2 levels) in <main>'
# ------------------
# --- Caused by: ---
# PG::SREModifyingSqlDataNotPermitted:
# ERROR: Table: "ci_pipelines" is write protected within this Gitlab database.
# HINT: Make sure you are using the right database connection
# CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
4) ScheduleFixingSecurityScanStatuses#down deletes all batched migration records
Failure/Error: connection.public_send(...)
ActiveRecord::StatementInvalid:
PG::SREModifyingSqlDataNotPermitted: ERROR: Table: "ci_pipelines" is write protected within this Gitlab database.
HINT: Make sure you are using the right database connection
CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
# ./spec/migrations/schedule_fixing_security_scan_statuses_spec.rb:17:in `block (2 levels) in <main>'
# ./spec/spec_helper.rb:506:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:498:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:494:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:494:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:242:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/migration.rb:53:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:108:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:62:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:108:in `block (2 levels) in <main>'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:6:in `block in with_gitlab_schemas_validate_connection_prevented'
# ./lib/gitlab/database/query_analyzers/base.rb:39:in `with_suppressed'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:5:in `with_gitlab_schemas_validate_connection_prevented'
# ./spec/support/database/gitlab_schemas_validate_connection.rb:11:in `block (2 levels) in <main>'
# ------------------
# --- Caused by: ---
# PG::SREModifyingSqlDataNotPermitted:
# ERROR: Table: "ci_pipelines" is write protected within this Gitlab database.
# HINT: Make sure you are using the right database connection
# CONTEXT: PL/pgSQL function gitlab_schema_prevent_write() line 4 at RAISE
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send'
Finished in 48.05 seconds (files took 57.39 seconds to load)
4 examples, 4 failures
Failed examples:
rspec ./spec/migrations/schedule_fixing_security_scan_statuses_spec.rb:52 # ScheduleFixingSecurityScanStatuses#up when the migration does not run on GitLab.com or development environment does not schedule the migration
rspec './spec/migrations/schedule_fixing_security_scan_statuses_spec.rb[1:1:2:1:1]' # ScheduleFixingSecurityScanStatuses#up when the migration runs on GitLab.com behaves like scheduler for fixing the security scans status schedules background job
rspec './spec/migrations/schedule_fixing_security_scan_statuses_spec.rb[1:1:3:1:1]' # ScheduleFixingSecurityScanStatuses#up when the migration runs on dev environment behaves like scheduler for fixing the security scans status schedules background job
rspec ./spec/migrations/schedule_fixing_security_scan_statuses_spec.rb:71 # ScheduleFixingSecurityScanStatuses#down deletes all batched migration records
Randomized with seed 39250Steps to reproduce
Proposed Resolution
Please refer to the Resolution guidance to learn more about resolution of broken master.
Edited by David Dieulivol