Failed to upgrade postgres from 9.6 to 10.9 on selinux env

Summary

Postgres upgrade from 9.x to 10.x is not working in selinux docker with nas on NFSv3.

Steps to reproduce

In docker :
Upgrade from 12.1.6-ce.0 to 12.2.5-ce.0
/var/opt/gitlab is a mounted volume on a nas.
SElinux is activated but not enforced during testing.
Docker container is started with :

    privileged: true
    userns_mode: host

What is the current bug behavior?

The postgres upgrade failed because it cannot create the data.10 directory. It's reported as succesfull if we create the directory and set the correct right manually, but the migration isn't done

What is the expected correct behavior?

a correct migration

Relevant logs and/or screenshots

root@gitlab:/# gitlab-ctl pg-upgrade
Checking for an omnibus managed postgresql: OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 10.9
Checking if we already upgraded: NOT OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Checking if postgresql['version'] is set: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details

Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 1s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-monitor: 0s, normally up
ok: down: gitlab-pages: 0s, normally up
ok: down: grafana: 0s, normally up
ok: down: logrotate: 1s, normally up
ok: down: node-exporter: 0s, normally up
ok: down: postgres-exporter: 1s, normally up
ok: down: redis-exporter: 0s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: sshd: 0s, normally up
Toggling services: OK
Stopping the database:ok: down: postgresql: 0s, normally up
Stopping the database: OK
Symlink correct version of binaries: OK
Creating temporary data directory:Error creating new directory: /var/opt/gitlab/postgresql/data.10
STDOUT:
STDERR: setgid: Operation not permitted
Creating temporary data directory: NOT OK
== Fatal error ==
Please check the output
Traceback (most recent call last):
        9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
        8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
        7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
        6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
        5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
        4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:175:in `block in load_file'
        3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:228:in `general_upgrade'
        2: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:187:in `common_pre_upgrade'
        1: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:293:in `create_temp_data_dir'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:150:in `die': undefined local variable or method `revert' for GitlabCtl::PgUpgrade:Class (NameError)

It's reported as succesfull if we create the directory and set the correct right manually :

root@gitlab:/# mkdir /var/opt/gitlab/postgresql/data.10
root@gitlab:/# cd /var/opt/gitlab/postgresql/
root@gitlab:/var/opt/gitlab/postgresql# ls -al
total 24
drwxr-xr-x.  6 gitlab-psql gitlab-psql 4096 Sep 30 14:03 .
drwxr-xr-x. 23 root        root        4096 Sep 30 13:36 ..
-rw-------.  1 gitlab-psql gitlab-psql   52 Feb 20  2018 .profile
drwx------. 19 gitlab-psql gitlab-psql 4096 Sep 30 13:39 data
drwxr-xr-x.  2 root        root        4096 Sep 30 14:03 data.10
drwx------. 19 gitlab-psql gitlab-psql 4096 Sep 30 10:26 data.9.6.14
drwx------. 19 gitlab-psql gitlab-psql 4096 Sep 30 10:25 data.bak
root@gitlab:/var/opt/gitlab/postgresql# chown gitlab-psql:gitlab-psql data.10
root@gitlab:/var/opt/gitlab/postgresql# chmod 700 data.10
root@gitlab:/var/opt/gitlab/postgresql# gitlab-ctl pg-upgrade
Checking for an omnibus managed postgresql: OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 10.9
Checking if we already upgraded: OK
The latest version 10.9 is already running, nothing to do
root@gitlab:/var/opt/gitlab/postgresql# ls data.10/
root@gitlab:/var/opt/gitlab/postgresql# cat data/PG_VERSION
9.6

Results of GitLab environment info

Expand for output related to GitLab environment info

after a restart of the container, since the db part was down :

root@gitlab:/# gitlab-rake gitlab:env:info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.6.3p62
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.22.0
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        12.2.5
Revision:       09f8edbc29a
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.14
URL:            https://url.com
HTTP Clone URL: https://url.com/some-group/some-project.git
SSH Clone URL:  ssh://git@amp-uat-repo-sources.equant.com:2222/some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        9.3.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

root@gitlab:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/2 ... yes
1/3 ... yes
9/4 ... yes
11/8 ... yes
11/10 ... yes
18/15 ... yes
9/19 ... yes
19/20 ... yes
51/25 ... yes
19/27 ... yes
19/30 ... yes
54/32 ... yes
19/40 ... yes
19/41 ... yes
19/56 ... yes
72/57 ... yes
73/58 ... yes
74/59 ... yes
73/60 ... yes
75/61 ... yes
75/62 ... yes
75/63 ... yes
75/64 ... yes
27/65 ... yes
27/66 ... yes
27/67 ... yes
78/68 ... yes
98/70 ... yes
104/73 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.0)
Git user has default SSH configuration? ... yes
Active users: ... 27
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

I haven't been able to pinpoint where the folder is created and how.