Gitlab helm chart always fails to install with database errors

Trying from 2 months. Unable to install gitlab helm chart on kubernetes clusters. Tried on 3 different kubernetes clusters. characteristics of our clusters

  • On prem
  • baremetal
  • Created and managed by Rancher 2.x
  • NFS as default storage class
  • Metallb with available IP addresses for charts to consume.

First and most prominent problem: External postgresql never connects and internal postgresql always fails with following error.

Let us for not talk only about all internal stuff with only mandatory parameters.

certmanager-issuer.email   =  memail@mycompany.com
global.edition=ce 
$ kubectl -n gitlab logs gitlab-postgresql-0 -c init-chmod-data
chown: changing ownership of '/bitnami/postgresql/data': Operation not permitted
chown: changing ownership of '/bitnami/postgresql': Operation not permitted
chown: changing ownership of '/bitnami/postgresql/data': Operation not permitted
$ kubectl -n gitlab logs gitlab-postgresql-0 -c metrics
time="2020-04-03T19:51:10Z" level=info msg="Established new database connection to \"127.0.0.1:5432\"." source="postgres_exporter.go:814"
time="2020-04-03T19:51:11Z" level=info msg="Established new database connection to \"127.0.0.1:5432\"." source="postgres_exporter.go:814"
time="2020-04-03T19:51:13Z" level=info msg="Established new database connection to \"127.0.0.1:5432\"." source="postgres_exporter.go:814"
time="2020-04-03T19:51:16Z" level=error msg="Error opening connection to database (postgresql://gitlab:PASSWORD_REMOVED@127.0.0.1:5432/gitlabhq_production?sslmode=disable): dial tcp 127.0.0.1:5432: connect: connection refused" source="postgres_exporter.go:1403"
time="2020-04-03T19:51:16Z" level=info msg="Starting Server: :9187" source="postgres_exporter.go:1576"
time="2020-04-03T19:51:52Z" level=info msg="Established new database connection to \"127.0.0.1:5432\"." source="postgres_exporter.go:814"
time="2020-04-03T19:51:53Z" level=info msg="Established new database connection to \"127.0.0.1:5432\"." source="postgres_exporter.go:814"
time="2020-04-03T19:51:55Z" level=info msg="Established new database connection to \"127.0.0.1:5432\"." source="postgres_exporter.go:814"
time="2020-04-03T19:51:58Z" level=error msg="Error opening connection to database (postgresql://gitlab:PASSWORD_REMOVED@127.0.0.1:5432/gitlabhq_production?sslmode=disable): dial tcp 127.0.0.1:5432: connect: connection refused" source="postgres_exporter.go:1403"

It looks like the bitnami postgresql db that you are using is not able to set permissions or something. We have a bitnami/postgresql chart deployed and working and I wish instead of you asking us to use "unencoded" (I know coded, encoded, decoded, unecrypted but do not know what is unencoded). Your example secret creation is very confusing since it creates two passwords but the section of the documentation about external database uses only one of those keys I would rather copy paste a connection string of the existing postgresql or directly copy paste, username, password, host and you at the backend go ahead and create the secret in whichever format (unencoded - i am still confused) you want and use it.

Edited by Jason Plum