gitlab-rake geo:db:refresh_foreign_tables fails on app server in HA setup after upgrade to 12.1.9
Summary
After the upgrade of an HA Geo cluster from 12.0.9 to 12.1.9 the FDW tables are not refreshed. This issue is known; however, running gitlab-rake geo:db:refresh_foreign_tables
on the secondary app server to refresh the FDW tables fails on the first try. Restarting the geo tracking database fixed the issue and rerunning was successful. It is unclear if we could have just waited a bit longer.
Steps to reproduce
(How one can reproduce the issue - this is very important)
- Follow steps as described in omnibus-gitlab!3577 (merged) These steps are new and we never had any official documentation for how to upgrade HA + Geo.
- After all steps are complete run
sudo gitlab-rake gitlab:geo:check
on secondary app-01 server
> sudo gitlab-rake gitlab:geo:check
Checking Geo ...
GitLab Geo is available ... yes
GitLab Geo is enabled ... yes
GitLab Geo secondary database is correctly configured ... yes
Database replication enabled? ... yes
Database replication working? ... yes
GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes
GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... no
Try fixing it:
Follow Geo setup instructions to configure secondary nodes with FDW support
If you upgraded recently check for any new step required to enable FDW
If you are using Omnibus GitLab try running:
gitlab-ctl reconfigure
If installing from source, try running:
bundle exec rake geo:db:refresh_foreign_tables
For more information see:
doc/gitlab-geo/database.md
GitLab Geo HTTP(S) connectivity ...
* Can connect to the primary node ... yes
HTTP/HTTPS repository cloning is enabled ... yes
Machine clock is synchronized ... yes
Git user has default SSH configuration? ... yes
OpenSSH configured to use AuthorizedKeysCommand ... yes
GitLab configured to disable writing to authorized_keys file ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking Geo ... Finished
- Run
sudo gitlab-rake geo:db:refresh_foreign_tables
Refreshing foreign tables for FDW: gitlab_secondary ... rake aborted!
ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
: ROLLBACK
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
: DROP SCHEMA IF EXISTS gitlab_secondary CASCADE;
CREATE SCHEMA gitlab_secondary;
IMPORT FOREIGN SCHEMA public
FROM SERVER gitlab_secondary
INTO gitlab_secondary;
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:49:in `block (2 levels) in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:49:in `block (2 levels) in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => geo:db:refresh_foreign_tables
(See full trace by running task with --trace)
- Then tried to select information schema in foreign tables on tracking DB node and got transient error:
gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables;
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Succeeded.
This worked after running again
- Restarting the tracking database resolved the error
Example Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)
What is the current bug behavior?
(What actually happens)
sudo gitlab-rake geo:db:refresh_foreign_tables
fails on secondary ha app server
What is the expected correct behavior?
(What you should see instead)
sudo gitlab-rake geo:db:refresh_foreign_tables
finishes successfully.
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)
See above
Environment info
Primary (US)
VM Instance |
---|
ha-postgresql |
ha-redis |
ha-gitaly |
ha-app-01 |
ha-app-02 |
ha-load-balancer |
Secondary (EU)
VM Instance |
---|
geo-postgresql |
geo-tracking |
geo-redis |
geo-gitaly |
geo-app-01 |
geo-app-02 |
geo-load-balancer |
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
- Restart the tracking database
- Wait