PostgreSQL install prefix shouldn't include minor version number
When running an omnibus upgrade the installer removes the old files which includes the old Postgres in
/opt/gitlab/embedded/postgresql/9.6.5. The new files are located in
/opt/gitlab/embedded/postgresql/9.6.9 but the old binary doesn't look there. As a result when the upgrade happens users get errors about being unable to find data files such as timezone files, extensions, or encoding libraries. Eventually they would restart the server to get the new binaries and the problems go away. To avoid errors they would have to shut down the server before doing the upgrade and do failovers during the upgrade.
When installing a new minor version of Postgres such as 9.6.8 over 9.6.5 the usual way binary distributions and packages such as the Debian packages or the Yum packages is to use a prefix such as
/usr/lib/postgres/9.6/. That means they replace the binaries and data files in place.
The running process (which is still the old binary) is supposed to still be able to read the data files and load the shared library files from the newer version as they're supposed to be ABI compatible. The release notes will highlight if for critical security issues this may not be true. This ensures that the running binary can be left in running for the duration of the upgrade. Only when the files are all replaced does it have to be restarted to get the new binary running.
For an HA environment, if users are aware of the problem and shut down the servers for the duration of the upgrade, then this amounts to more or less the same thing as even a brief shutdown triggers a failover and the combined system is always available. But for a single-database environment this makes the outage for the upgrade substantially longer. For geo this was a particular problem as it's quite likely that not every database (every site and tracking database) is set up as a HA environment.
Incidentally, to do this it is important that the installer use the equivalent of
mv to install files. That is to create the files in a temporary name and then rename them over the old file. This is definitely the way
rpm install files and also the way the
install binary used by most make script works.