Geo upgrade from 12.0 to 12.1 fails on the secondary via Omnibus
Summary
When upgrading a simple Geo installation with a single Primary node and a single secondary the upgrade from 12.0.8 to 12.1.8 fails on the secondary.
Steps to reproduce
- Upgrade Gitlab on primary and secondary from 11.11.5 to 12.0.8 using general update steps
- Upgrade Gitlab on primary 12.0.8 to 12.1.8
- After
sudo apt-get install gitlab-ee=12.1.8-ee.0
on the secondary
[...]
Running handlers complete
Chef Client failed. 10 resources updated in 01 minutes 09 seconds
Ensuring PostgreSQL is updated:Traceback (most recent call last):
8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>
'
5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
4: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add
_command_under_category'
3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:79:in `block in load_file'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:105:in `roles'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:64:in `get_node_attributes'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:52:in `parse_json_file': Attributes not found in /opt/g
itlab/embedded/nodes/fzgeo-11115-down-secondary-geo.c.group-geo-f9c951.internal.json, has reconfigure been run yet? (Gi
tlabCtl::Errors::NodeError)
Ensuring PostgreSQL is updated: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
dpkg: error processing package gitlab-ee (--configure):
installed gitlab-ee package post-installation script subprocess returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)
>sudo cat /opt/gitlab/embedded/nodes/fzgeo-11115-test-secondary-geo.c.group-geo-f9c951.internal.json
{
"name": "fzgeo-11115-test-secondary-geo.c.group-geo-f9c951.internal"
}
What is the current bug behavior?
(What actually happens)
- GitLab Omnibus install fails on the secondary
What is the expected correct behavior?
(What you should see instead)
- GitLab Omnibus install succeeds on the secondary
Relevant logs and/or screenshots
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
)
Results of GitLab application Check
sudo gitlab-rake gitlab:check SANITIZE=true
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transactio
n and exit, because another server process exited abnormally and possibly corrupted shared me
mory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
rake aborted!
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connect
ion unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
: SELECT a.attname, format_type(a.atttypid, a.atttypmod),
pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
c.collname, col_description(a.attrelid, a.attnum) AS comment
FROM pg_attribute a
LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
LEFT JOIN pg_type t ON a.atttypid = t.oid
LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.ty
pcollation
WHERE a.attrelid = '"namespaces"'::regclass
AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/group.rb:70:in `block in <module:G
roup>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/patch/prependable.rb:33:in `class_eval'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/patch/prependable.rb:33:in `prepend_feat
ures'
/opt/gitlab/embedded/service/gitlab-rails/app/models/group.rb:452:in `prepend'
/opt/gitlab/embedded/service/gitlab-rails/app/models/group.rb:452:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/entities.rb:370:in `<class:Group>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/entities.rb:361:in `<module:Entities>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/entities.rb:4:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/entities.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:18:in `block (3 levels)
in <class:AccessRequests>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:16:in `block (2 levels)
in <class:AccessRequests>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:15:in `block in <class:A
ccessRequests>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:11:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:11:in `<class:AccessRequ
ests>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:4:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/access_requests.rb:3:in `<top (required)>'
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
I managed to finish the upgrade using this sequence of steps:
> sudo gitlab-ctl reconfigure
> sudo apt-get install gitlab-ee=12.1.8-ee.0
[...]
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:52:in `parse_json_file': Attributes not found in /opt/g
itlab/embedded/nodes/fzgeo-11115-down-secondary-geo.c.group-geo-f9c951.internal.json, has reconfigure been run yet? (Gi
tlabCtl::Errors::NodeError)
Ensuring PostgreSQL is updated: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
dpkg: error processing package gitlab-ee (--configure):
installed gitlab-ee package post-installation script subprocess returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)
> sudo gitlab-ctl restart postgresql
> sudo gitlab-ctl reconfigure
--> Success
> sudo gitlab-ctl restart geo-postgresql
--> Success
> sudo gitlab-ctl pg-upgrade
> sudo gitlab-ctl replicate-geo-database --slot-name=singleslot --host=35.246.184.