Skip to content

Geo: Improve pg-upgrade procedure doc

Items found when testing pg-upgrade with Geo: #4975 (closed).

  • Edit the following text “Note: In a Geo HA setup with databases managed by GitLab Omnibus, the command above should be run on your Geo tracking database.” to read:
    • “Note: In a Geo HA setup with databases managed by GitLab Omnibus, the command above should be run on an app node in the secondary.”
  • Modify “Database settings > Upgrading a Geo cluster” doc to:
    • Note how to specify the target PG version?
    • Modify “Find the existing name of the replication slot name on the primary node, run:” to say that it needs to be run on the primary DB node
  • We could clarify the beginning of the section https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance better. There are multiple distinct things we are trying to say, all at once. Who is reading this? What are they trying to do? How can we guide them to success?
  • With each PG DB on its own node, after the replicate-geo-database step, you need to gitlab-ctl restart the read-replica PG node or else you'll get this error when refreshing foreign tables:
mkozono-ansible@mkozono-ha2-omnibus4838-secondary-geo:~$ sudo gitlab-rake geo:db:refresh_foreign_tables

Refreshing foreign tables for FDW: gitlab_secondary ... rake aborted!
ActiveRecord::StatementInvalid: PG::SqlclientUnableToEstablishSqlconnection: ERROR:  could not connect to server "gitlab_secondary"
DETAIL:  SSL error: certificate verify failed
FATAL:  no pg_hba.conf entry for host "10.138.0.59", user "gitlab", database "gitlabhq_production", SSL off
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:49:in `block (2 levels) in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::SqlclientUnableToEstablishSqlconnection: ERROR:  could not connect to server "gitlab_secondary"
DETAIL:  SSL error: certificate verify failed
FATAL:  no pg_hba.conf entry for host "10.138.0.59", user "gitlab", database "gitlabhq_production", SSL off
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:49:in `block (2 levels) in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:48:in `block in refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/database_tasks.rb:185:in `with_geo_db'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/geo/geo_tasks.rb:47:in `refresh_foreign_tables!'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => geo:db:refresh_foreign_tables
(See full trace by running task with --trace)
Edited by Michael Kozono