Gitlab GEO: Error while promoting old primary (secondary now) as primary again.
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Using gitlab with docker.
I already had performed failover to secondary.
Now I was bringing back the old primary that is as secondary now. I put primary offline. Paused the replication on secondary with gitlab-ctl geo-replication-pause
Started promoting secondary to primary but got the error below, why this happened? After this error I started replication again fearing that was because the pausing and got a new error.
root@gitlab:/# gitlab-ctl geo-replication-pause
* Pausing replication
* Replication paused
* Create Geo point-in-time recovery file
root@gitlab:/#
root@gitlab:/#
root@gitlab:/# gitlab-ctl geo promote
WARNING: The current secondary node will now be promoted to a primary node. Are you sure you want to proceed? (y/n)
y
Checking if we need to promote any service running on this node: OK
Detected a PostgreSQL Standby server cluster.
Promoting the PostgreSQL to end standby mode and begin read-write operations:
Recovery to point 56/C6A1F6E8 and promoting...
ok: run: postgresql: (pid 345002) 0s
The database is successfully promoted!
Promoting the PostgreSQL to end standby mode and begin read-write operations: OK
Detected an application or a Sidekiq or a Geo log cursor or a Geo PostgreSQL node.
Disabling the secondary services and enabling the primary services in the cluster configuration file:/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/ffi-yajl-2.6.0/lib/ffi_yajl/encoder.rb:42: warning: undefining the allocator of T_DATA class FFI_Yajl::Ext::Encoder::YajlGen
Disabling the secondary services and enabling the primary services in the cluster configuration file: OK
Attempting to detect the role of this Geo node:
root@gitlab:/# gitlab-ctl geo promote
WARNING: The current secondary node will now be promoted to a primary node. Are you sure you want to proceed? (y/n)
y
Checking if we need to promote any service running on this node: OK
Detected a PostgreSQL Standby server cluster.
Promoting the PostgreSQL to end standby mode and begin read-write operations:
Recovery to point 56/C6A1F6E8 and promoting...
ok: run: postgresql: (pid 345002) 0s
The database is successfully promoted!
Promoting the PostgreSQL to end standby mode and begin read-write operations: OK
Detected an application or a Sidekiq or a Geo log cursor or a Geo PostgreSQL node.
Disabling the secondary services and enabling the primary services in the cluster configuration file:/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/ffi-yajl-2.6.0/lib/ffi_yajl/encoder.rb:42: warning: undefining the allocator of T_DATA class FFI_Yajl::Ext::Encoder::YajlGen
Disabling the secondary services and enabling the primary services in the cluster configuration file: OK
Attempting to detect the role of this Geo node:secondary
Attempting to detect the role of this Geo node: OK
Detected an application node.
Promoting secondary site to primary site:rake aborted!
ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute DELETE in a read-only transaction
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:31:in `block in set_secondary_as_primary'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:91:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:399:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:398:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:82:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:21:in `set_secondary_as_primary'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:83:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
PG::ReadOnlySqlTransaction: ERROR: cannot execute DELETE in a read-only transaction
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:31:in `block in set_secondary_as_primary'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:91:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:399:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:398:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:82:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:21:in `set_secondary_as_primary'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:83:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => geo:set_secondary_as_primary
(See full trace by running task with --trace)
Promoting secondary site to primary site: NOT OK
Unable to promote secondary site to primary site.
root@gitlab:/# ^C
root@gitlab:/# gitlab-ctl stop
`
Here the error that I got after I started replication even with primary down and started promote again:
`
root@gitlab:/# gitlab-ctl geo-replication-resume
* Resume replication
* Replication resumed
* Remove Geo point-in-time recovery file
root@gitlab:/# gitlab-ctl geo promote
WARNING: The current secondary node will now be promoted to a primary node. Are you sure you want to proceed? (y/n)
y
Checking if we need to promote any service running on this node: OK
Detected a PostgreSQL Standby server cluster.
Promoting the PostgreSQL to end standby mode and begin read-write operations:
Promoting the PostgreSQL read-only replica to primary...
waiting for server to promote............................................................... stopped waiting
pg_ctl: server did not promote in time
Error while promoting the current node: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/gitlab-pg-ctl promote ----
STDOUT: waiting for server to promote............................................................... stopped waiting
STDERR: pg_ctl: server did not promote in time
---- End output of /opt/gitlab/embedded/bin/gitlab-pg-ctl promote ----
Ran /opt/gitlab/embedded/bin/gitlab-pg-ctl promote returned 1
root@gitlab:/#
`
So I powered down the container and started again, gitlab server come up and tried promote again and the same error.
`
WARNING: The current secondary node will now be promoted to a primary node. Are you sure you want to proceed? (y/n)
y
Checking if we need to promote any service running on this node: OK
Detected a PostgreSQL Standby server cluster.
Promoting the PostgreSQL to end standby mode and begin read-write operations:
Promoting the PostgreSQL read-only replica to primary...
waiting for server to promote............................................................... stopped waiting
pg_ctl: server did not promote in time
Error while promoting the current node: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/gitlab-pg-ctl promote ----
STDOUT: waiting for server to promote............................................................... stopped waiting
STDERR: pg_ctl: server did not promote in time
---- End output of /opt/gitlab/embedded/bin/gitlab-pg-ctl promote ----
Ran /opt/gitlab/embedded/bin/gitlab-pg-ctl promote returned 1
root@gitlab:/#
`Edited by 🤖 GitLab Bot 🤖