Geo upgrade from 12.0 to 12.1 fails on the secondary via Omnibus

Summary

When upgrading a simple Geo installation with a single Primary node and a single secondary the upgrade from 12.0.8 to 12.1.8 fails on the secondary.

Steps to reproduce

Upgrade Gitlab on primary and secondary from 11.11.5 to 12.0.8 using general update steps
Upgrade Gitlab on primary 12.0.8 to 12.1.8
After sudo apt-get install gitlab-ee=12.1.8-ee.0 on the secondary

[...]
Running handlers complete
Chef Client failed. 10 resources updated in 01 minutes 09 seconds
Ensuring PostgreSQL is updated:Traceback (most recent call last):
        8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
        7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
        6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>
'
        5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
        4: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add
_command_under_category'
        3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:79:in `block in load_file'
        2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:105:in `roles'
        1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:64:in `get_node_attributes'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:52:in `parse_json_file': Attributes not found in /opt/g
itlab/embedded/nodes/fzgeo-11115-down-secondary-geo.c.group-geo-f9c951.internal.json, has reconfigure been run yet? (Gi
tlabCtl::Errors::NodeError)
Ensuring PostgreSQL is updated: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
dpkg: error processing package gitlab-ee (--configure):
 installed gitlab-ee package post-installation script subprocess returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)

>sudo cat /opt/gitlab/embedded/nodes/fzgeo-11115-test-secondary-geo.c.group-geo-f9c951.internal.json
{
  "name": "fzgeo-11115-test-secondary-geo.c.group-geo-f9c951.internal"
}

What is the current bug behavior?

(What actually happens)

GitLab Omnibus install fails on the secondary

What is the expected correct behavior?

(What you should see instead)

GitLab Omnibus install succeeds on the secondary

Relevant logs and/or screenshots

2019-09-11-apt.log

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Results of GitLab application Check

sudo gitlab-rake gitlab:check SANITIZE=true
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transactio
n and exit, because another server process exited abnormally and possibly corrupted shared me
mory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
rake aborted!
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connect
ion unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
                     c.collname, col_description(a.attrelid, a.attnum) AS comment
                FROM pg_attribute a
                LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
                LEFT JOIN pg_type t ON a.atttypid = t.oid
                LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.ty
pcollation
               WHERE a.attrelid = '"namespaces"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/group.rb:70:in `block in <module:G
roup>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/patch/prependable.rb:33:in `class_eval'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/patch/prependable.rb:33:in `prepend_feat
ures'
/opt/gitlab/embedded/service/gitlab-rails/app/models/group.rb:452:in `prepend'
/opt/gitlab/embedded/service/gitlab-rails/app/models/group.rb:452:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/entities.rb:370:in `<class:Group>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/entities.rb:361:in `<module:Entities>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/entities.rb:4:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/entities.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:18:in `block (3 levels) 
in <class:AccessRequests>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:16:in `block (2 levels) 
in <class:AccessRequests>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:15:in `block in <class:A
ccessRequests>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:11:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:11:in `<class:AccessRequ
ests>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:4:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:3:in `<top (required)>'

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

I managed to finish the upgrade using this sequence of steps:

> sudo gitlab-ctl reconfigure
> sudo apt-get install gitlab-ee=12.1.8-ee.0
[...]
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:52:in `parse_json_file': Attributes not found in /opt/g
itlab/embedded/nodes/fzgeo-11115-down-secondary-geo.c.group-geo-f9c951.internal.json, has reconfigure been run yet? (Gi
tlabCtl::Errors::NodeError)
Ensuring PostgreSQL is updated: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
dpkg: error processing package gitlab-ee (--configure):
 installed gitlab-ee package post-installation script subprocess returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)
> sudo gitlab-ctl restart postgresql
> sudo gitlab-ctl reconfigure
--> Success
> sudo gitlab-ctl restart geo-postgresql
--> Success
> sudo gitlab-ctl pg-upgrade
> sudo gitlab-ctl replicate-geo-database --slot-name=singleslot --host=35.246.184.

Edited Sep 18, 2019 by Fabian Zimmer