Skip to content

gitlab-rake geo:db:refresh_foreign_tables fails on app server in HA setup after upgrade to 12.1.9

Summary

After the upgrade of an HA Geo cluster from 12.0.9 to 12.1.9 the FDW tables are not refreshed. This issue is known; however, running gitlab-rake geo:db:refresh_foreign_tables on the secondary app server to refresh the FDW tables fails on the first try. Restarting the geo tracking database fixed the issue and rerunning was successful. It is unclear if we could have just waited a bit longer.

Steps to reproduce

(How one can reproduce the issue - this is very important)

  • Follow steps as described in omnibus-gitlab!3577 (merged) These steps are new and we never had any official documentation for how to upgrade HA + Geo.
  • After all steps are complete run sudo gitlab-rake gitlab:geo:check on secondary app-01 server
> sudo gitlab-rake gitlab:geo:check
Checking Geo ...


GitLab Geo is available ... yes
GitLab Geo is enabled ... yes
GitLab Geo secondary database is correctly configured ... yes
Database replication enabled? ... yes
Database replication working? ... yes
GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes
GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... no
  Try fixing it:
  Follow Geo setup instructions to configure secondary nodes with FDW support
  If you upgraded recently check for any new step required to enable FDW
  If you are using Omnibus GitLab try running:
  gitlab-ctl reconfigure
  If installing from source, try running:
  bundle exec rake geo:db:refresh_foreign_tables
  For more information see:
  doc/gitlab-geo/database.md
GitLab Geo HTTP(S) connectivity ...
* Can connect to the primary node ... yes
HTTP/HTTPS repository cloning is enabled ... yes
Machine clock is synchronized ... yes
Git user has default SSH configuration? ... yes
OpenSSH configured to use AuthorizedKeysCommand ... yes
GitLab configured to disable writing to authorized_keys file ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking Geo ... Finished
  • Run sudo gitlab-rake geo:db:refresh_foreign_tables 
Refreshing foreign tables for FDW: gitlab_secondary ... rake aborted!
ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
: ROLLBACK
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'


Caused by:
PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'


Caused by:
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
: DROP SCHEMA IF EXISTS gitlab_secondary CASCADE;
CREATE SCHEMA gitlab_secondary;
IMPORT FOREIGN SCHEMA public
  FROM SERVER gitlab_secondary
  INTO gitlab_secondary;
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:49:in `block (2 levels) in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'


Caused by:
PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:49:in `block (2 levels) in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => geo:db:refresh_foreign_tables
(See full trace by running task with --trace)
  • Then tried to select information schema in foreign tables on tracking DB node and got transient error:
gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables;
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
The connection to the server was lost. Attempting reset: Succeeded.

This worked after running again

  • Restarting the tracking database resolved the error

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)

What is the current bug behavior?

(What actually happens)

sudo gitlab-rake geo:db:refresh_foreign_tables fails on secondary ha app server

What is the expected correct behavior?

(What you should see instead)

sudo gitlab-rake geo:db:refresh_foreign_tables finishes successfully.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

See above

Environment info

Primary (US)

VM Instance
ha-postgresql
ha-redis
ha-gitaly
ha-app-01
ha-app-02
ha-load-balancer

Secondary (EU)

VM Instance
geo-postgresql
geo-tracking
geo-redis
geo-gitaly
geo-app-01
geo-app-02
geo-load-balancer

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

  • Restart the tracking database
  • Wait