Skip to content

Fresh GEO instance not working with 15.10 - solution included

I want to share my experience with installing a fresh Gitlab-ee 15.10 using Docker to be used as GEO.

TL;DR: upgrade the Primary to Postgresql 13 before starting with GEO using gitlab-ctl pg_upgrade.

Fresh install

When following the instructions, I ended up with:

The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 12.12

Seemingly geo-postgresql was 12.12, while postgresql was 13.8. Replication was using 13.8. Not sure what was going on...

When I threw away the data-dir and started over with the edited gitlab.rb, I got:

The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 13.8

Now replication was using 12.12 still, but geo-postgresql and postgresql were 13.8. Like the reverse is going on.

For extra fun 12.12 uses wal_keep_segments and 13.8 uses wal_keep_size - neither recognized the other. This has caused a lot of problems, as I had to update the config-file continuously!

Below sums up what I've tried, to upgrade 12.12 to 13.8 - don't try this yourself, and read the last part. This is for emergency situations when gitlab-ctl pg_upgrade does not work, as not much documentation is on this subject.

docker exec -it --user gitlab-psql gitlab_main_1 bash
pg_ctl stop -D /var/opt/gitlab/postgresql/data
/opt/gitlab/embedded/postgresql/12/bin/pg_ctl start -D /var/opt/gitlab/postgresql/data
/opt/gitlab/embedded/postgresql/12/bin/pg_ctl promote -D /var/opt/gitlab/postgresql/data
initdb /var/opt/gitlab/postgresql/data_13/
/opt/gitlab/embedded/postgresql/12/bin/pg_ctl stop -D /var/opt/gitlab/postgresql/data
cd /tmp
pg_upgrade -d /var/opt/gitlab/postgresql/data -D /var/opt/gitlab/postgresql/data_13 -b /opt/gitlab/embedded/postgresql/12/bin -B /opt/gitlab/embedded/postgresql/13/bin -v

# rm -rf '/var/opt/gitlab/postgresql/data'
# mv /var/opt/gitlab/postgresql/data_13 /var/opt/gitlab/postgresql/data

But after starting the replication, it backed up the existing database and then initiated the database with 12.12. There was no reference to /opt/gitlab/embedded/postgresql/12/bin/initdb anywhere, so I saw no reason to keep thinking in this direction.

I gave up and tried to try the upgrade route, to make sure all was Postgresql 12.

Upgrade from 14.10

I set the version to "14.10.5-ee.0" to make sure Postgresql 12.7 would be installed. A fresh install, but I had to run gitlab-rake db:migrate and then restart the services after it started the first time, to fix all kinds of database-errors, like:

PG::UndefinedTable: ERROR: relation "application_settings" does not exist (ActiveRecord::StatementInvalid)

and loads of other missing stuff and tables. Ugh!

Seemingly it does not work well, when the application is started for the first time with a gitlab.rb configured for Geo? This would make it very difficult to quickly add a new secondary, as it's then designed to be manual work.

Anyway, it seems to work with a bit of hacking and weirdness, with these two highlights:

  • Had to increase max_connections to be the same or higher as the master DB.
  • Had to wait for an hour, till the status moved from "Unknown" to "Healthy". (???) Seemingly it was already syncing, but nothing was shown.

After that I upgraded to 15.0.5-ee.0 and after it was running, I upgraded to latest. The secondary is now shown as healthy.

And then I realized what was the reason

At the end of day two, while I gave myself 4 hours...

The primary uses Postgresql 12.12, as it followed the upgrade path. Seemingly, when replicating, the database is bit-copied and not content-copied - this explains the "The data directory was initialized by PostgreSQL version 12". This does not explain why it was the reverse the first time.

So what needs to be done to get this fixed?

Preferred is that Gitlab Geo checks the database-version, before proceeding with the replication.

But also documentation could help a lot. There are several issues already the documentation for GEO is outdated or plain wrong - it seems that new functionality is added to Geo, but the documentation still requests these things to be done manually. Also all info for Docker is missing. But to focus on this problem: please add that the user has to check what is the version of the database and make sure it's the same on both primary and secondary.

cat /var/opt/gitlab/postgresql/data/PG_VERSION

Yes, it's open source... I'll see what I can do myself here)