Upgrading PostgreSQL to 10.7 fails due to locale issues

Summary

Trying to upgrade the PostgreSQL from 9.6 to 10.7 fails due to different lc_ctype settings of the old and new postgres database.

Steps to reproduce

Install GitLab Omnibus CE 11.1.4 on a system with LC_CTYPE=en_US.utf8
Upgrade the system in multiple steps to CE 11.11.1
Run gitlab-ctl reconfigure (as was suggested in other reports where the upgrade fails)
Run gitlab-ctl pg-upgrade

What is the current bug behavior?

The PostgreSQL upgrade fails due to differing lc_ctype settings of the old and new postgres database and PostgreSQL is reverted to 9.6.

Error message: lc_ctype values for database "postgres" do not match: old "en_US.utf8", new "C"

What is the expected correct behavior?

The PostgreSQL upgrade should finish successfully and should be upgraded to 10.7.

Relevant logs

gitlab [~]# gitlab-ctl pg-upgrade Checking for an omnibus managed postgresql: OK Checking for a newer version of PostgreSQL to install Upgrading PostgreSQL to 10.7 Checking if we already upgraded: NOT OK Checking if PostgreSQL bin files are symlinked to the expected location: OK Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.

Please hit Ctrl-C now if you want to cancel the operation. Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html Toggling deploy page: OK Toggling services:ok: down: alertmanager: 1s, normally up ok: down: gitaly: 0s, normally up ok: down: gitlab-monitor: 1s, normally up ok: down: logrotate: 0s, normally up ok: down: node-exporter: 0s, normally up ok: down: postgres-exporter: 0s, normally up ok: down: prometheus: 0s, normally up ok: down: redis-exporter: 1s, normally up ok: down: sidekiq: 0s, normally up Toggling services: OK Stopping the database:ok: down: postgresql: 1s, normally up Stopping the database: OK Symlink correct version of binaries: OK Creating temporary data directory: OK Initializing the new database: OK Upgrading the data:Error upgrading the data to version 10.7 STDOUT: Performing Consistency Checks

Checking cluster versions ok Checking database user is the install user ok Checking database connection settings ok Checking for prepared transactions ok Checking for reg* data types in user tables ok Checking for contrib/isn with bigint-passing mismatch ok Checking for invalid "unknown" user columns ok Creating dump of global objects ok Creating dump of database schemas gitlabhq_production postgres template1 ok

lc_ctype values for database "postgres" do not match: old "en_US.utf8", new "C" Failure, exiting STDERR: Upgrading the data: NOT OK == Fatal error == Error upgrading the database == Reverting == ok: down: postgresql: 6s, normally up Symlink correct version of binaries: OK ok: run: postgresql: (pid 4476) 1s == Reverted == == Reverted to 9.6.11. Please check output for what went wrong == Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html Toggling deploy page: OK Toggling services:ok: run: alertmanager: (pid 4491) 0s ok: run: gitaly: (pid 4503) 0s ok: run: gitlab-monitor: (pid 4526) 1s ok: run: logrotate: (pid 4541) 0s ok: run: node-exporter: (pid 4553) 1s ok: run: postgres-exporter: (pid 4560) 0s ok: run: prometheus: (pid 4570) 0s ok: run: redis-exporter: (pid 4661) 1s ok: run: sidekiq: (pid 4668) 0s Toggling services: OK

gitlab [~]# locale LANG=C LANGUAGE=en_US:en LC_CTYPE=en_US.utf8 LC_NUMERIC="C" LC_TIME="C" LC_COLLATE="C" LC_MONETARY="C" LC_MESSAGES="C" LC_PAPER="C" LC_NAME="C" LC_ADDRESS="C" LC_TELEPHONE="C" LC_MEASUREMENT="C" LC_IDENTIFICATION="C" LC_ALL=

Details of package version

Provide the package version installation details

gitlab [~]# dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                              Version               Architecture          Description
+++-=================================-=====================-=====================-=======================================================================
ii  gitlab-ce                         11.11.1-ce.0          amd64                 GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                                                         (no description available)

Environment details

Operating System: Debian 9.9 on amd64
Installation Target, remove incorrect values:
- VM: VMware ESX
Installation Type, remove incorrect values:
- Upgrade from version 11.11.0-ce.0
Is this a single or multiple node installation? single
Resources
- CPU: 4 cores, Intel Xeon E3-12xx v2 in VM host
- Memory total: 8G

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

external_url 'https://gitlab.sanitized'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@sanitized'
gitlab_rails['gitlab_default_can_create_group'] = true
gitlab_rails['gitlab_username_changing_enabled'] = false
gitlab_rails['trusted_proxies'] = [ 'sanitized' ]
gitlab_rails['usage_ping_enabled'] = false
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
  main: # 'main' is the GitLab 'provider ID' of this LDAP server
    label: 'LDAP'
    host: 'sanitized'
    port: 389
    uid: 'sAMAccountName'
    bind_dn: 'sanitized'
    password: 'sanitized'
    encryption: 'plain'
    verify_certificates: false
    active_directory: true
    allow_username_or_email_login: false
    lowercase_usernames: false
    block_auto_created_users: false
    base: 'sanitized'
    #user_filter: ''
    ## EE only
    group_base: ''
    admin_group: ''
    sync_ssh_keys: false
EOS
gitlab_rails['backup_path'] = "/net/sanitized/storage/gitlab-backup/data"
gitlab_rails['backup_keep_time'] = 2592000
gitlab_rails['gitlab_shell_ssh_port'] = 29418
nginx['listen_port'] = 80
nginx['listen_https'] = false