GitLab AutoDevOps integrated postgres readiness and liveliness probe fails
Summary
I am currently trying out create a pretty standard gitlab auto devops deployment with the shipped postgres chart.
Unfortunately it never comes live because of a failed Readiness and Liveliness probe.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned myproject-review-feat-deplo-k6mvj9/review-feat-deplo-k6mvj9-postgresql-0 to myproject1
Normal SuccessfulAttachVolume 79s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-160159a3-48e1-4b3d-b595-d04a3e1ac90a"
Normal Pulling 74s kubelet, myproject1 Pulling image "docker.io/bitnami/minideb:stretch"
Normal Pulled 73s kubelet, myproject1 Successfully pulled image "docker.io/bitnami/minideb:stretch"
Normal Created 73s kubelet, myproject1 Created container init-chmod-data
Normal Started 73s kubelet, myproject1 Started container init-chmod-data
Normal Pulling 72s kubelet, myproject1 Pulling image "docker.io/bitnami/postgresql:10"
Normal Pulled 63s kubelet, myproject1 Successfully pulled image "docker.io/bitnami/postgresql:10"
Normal Created 63s kubelet, myproject1 Created container review-feat-deplo-k6mvj9-postgresql
Normal Started 63s kubelet, myproject1 Started container review-feat-deplo-k6mvj9-postgresql
Warning Unhealthy 6s (x6 over 56s) kubelet, myproject1 Readiness probe failed: 127.0.0.1:5432 - no attempt
Warning Unhealthy 6s (x3 over 26s) kubelet, myproject1 Liveness probe failed: 127.0.0.1:5432 - no attempt
I shelled into the created postgres container and executed the readiness and liveliness probes manually which turned out fine (ret=0), interestingly enough it mentions "no attempt" as error message, which i failed to find out where this could come from.
Steps to reproduce
The project causing is confidential I'm afraid, but I think it's a pretty standard gitlab autodevops use case: These are my relevant ci configurations in the project: .gitlab-ci.yml
variables:
AUTO_DEVOPS_DEPLOY_DEBUG: 'true'
POSTGRES_ENABLED: 'true'
POSTGRES_VERSION: 10
AUTO_DEVOPS_POSTGRES_CHANNEL: 2
DB_MIGRATE: "python manage.py migrate"
include:
- template: Jobs/Build.gitlab-ci.yml
- template: Jobs/Deploy.gitlab-ci.yml
stages:
- test
- build
- review
- staging
- canary
- deploy
- incremental rollout 10%
- incremental rollout 25%
- incremental rollout 50%
- incremental rollout 100%
- production
- cleanup
frontend:lint:
stage: test
image: node:12
before_script:
- cd frontend/static_src
- npm i
script:
- npm run lint
cache:
paths:
- frontend/static_src/node_modules/
frontend:unit:
stage: test
image: node:12
before_script:
- cd frontend/static_src
- npm i
script:
- npm run test:unit
cache:
paths:
- frontend/static_src/node_modules/
frontend:e2e:
image: node:12-stretch
stage: test
before_script:
- wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
- apt update && apt install -y ./google-chrome-stable_current_amd64.deb
- cd frontend/static_src
- npm i
script:
- npm run test:e2e
cache:
paths:
- frontend/static_src/node_modules/
.gitlab/auto-devops-values.yaml
deploymentApiVersion: apps/v1
livenessProbe:
path: /app
readinessProbe:
path: /app
service:
externalPort: 8000
internalPort: 8000
Gitlab: 13.0.6-ee Kubernetes: RKE with kubernetes v1.17.6
Example Project
Can't share unfortunately
What is the current bug behavior?
Postgres with auto devops scripts never becomes available resulting in a failed deployment
What is the expected correct behavior?
Postgres becoming available
Relevant logs and/or screenshots
Logs from the created pg container:
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.78 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.78 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.78
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.82 INFO ==> ** Starting PostgreSQL setup **
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.84 INFO ==> Validating settings in POSTGRESQL_* env vars..
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.85 INFO ==> Loading custom pre-init scripts...
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.87 INFO ==> Initializing PostgreSQL database...
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.90 INFO ==> pg_hba.conf file not detected. Generating it...
Sat, Jun 20 2020 11:45:53 am
postgresql 09:45:53.90 INFO ==> Generating local authentication configuration
Sat, Jun 20 2020 11:45:55 am
postgresql 09:45:55.72 INFO ==> Starting PostgreSQL in background...
Sat, Jun 20 2020 11:45:55 am
postgresql 09:45:55.86 INFO ==> Changing password of postgres
Sat, Jun 20 2020 11:45:55 am
postgresql 09:45:55.91 INFO ==> Configuring replication parameters
Sat, Jun 20 2020 11:45:55 am
postgresql 09:45:55.94 INFO ==> Configuring fsync
Sat, Jun 20 2020 11:45:55 am
postgresql 09:45:55.95 INFO ==> Loading custom scripts...
Sat, Jun 20 2020 11:45:55 am
postgresql 09:45:55.96 INFO ==> Enabling remote connections
Sat, Jun 20 2020 11:45:55 am
postgresql 09:45:55.98 INFO ==> Stopping PostgreSQL...
Sat, Jun 20 2020 11:45:57 am
postgresql 09:45:57.00 INFO ==> ** PostgreSQL setup finished! **
Sat, Jun 20 2020 11:45:57 am
Sat, Jun 20 2020 11:45:57 am
postgresql 09:45:57.03 INFO ==> ** Starting PostgreSQL **
Sat, Jun 20 2020 11:45:57 am
2020-06-20 09:45:57.103 GMT [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
Sat, Jun 20 2020 11:45:57 am
2020-06-20 09:45:57.103 GMT [1] LOG: listening on IPv6 address "::", port 5432
Sat, Jun 20 2020 11:45:57 am
2020-06-20 09:45:57.111 GMT [1] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
Sat, Jun 20 2020 11:45:57 am
2020-06-20 09:45:57.150 GMT [103] LOG: database system was shut down at 2020-06-20 09:45:56 GMT
Sat, Jun 20 2020 11:45:57 am
2020-06-20 09:45:57.165 GMT [1] LOG: database system is ready to accept connections
Sat, Jun 20 2020 11:47:20 am
2020-06-20 09:47:20.304 GMT [1] LOG: received smart shutdown request
Sat, Jun 20 2020 11:47:20 am
2020-06-20 09:47:20.316 GMT [1] LOG: worker process: logical replication launcher (PID 109) exited with exit code 1
Sat, Jun 20 2020 11:47:20 am
2020-06-20 09:47:20.320 GMT [104] LOG: shutting down
Sat, Jun 20 2020 11:47:20 am
2020-06-20 09:47:20.366 GMT [1] LOG: database system is shut down
Note: It is shutdown again after readiness and liveliness fails. It's up for 2 minutes and seems to be fine though
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.26.2 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 13.0.6-ee Revision: 4c8f8b0d9ee Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: https://gitlab.alt.coop HTTP Clone URL: https://gitlab.alt.coop/some-group/some-project.git SSH Clone URL: git@gitlab.alt.coop:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: saml, google_oauth2 GitLab Shell Version: 13.2.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.2.0 ? ... OK (13.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/3 ... yes 3/8 ... yes 3/9 ... yes 2/10 ... yes 3/11 ... yes 4/13 ... yes 3/14 ... yes 2/15 ... yes 2/16 ... yes 6/18 ... yes 6/19 ... yes 2/20 ... yes 3/21 ... yes 3/22 ... yes 3/24 ... yes 9/25 ... yes 9/26 ... yes 10/28 ... yes 3/29 ... yes 3/31 ... yes 6/32 ... yes 14/33 ... yes 14/34 ... yes 14/35 ... yes 3/36 ... yes 4/37 ... yes 2/38 ... yes 13/39 ... yes 14/40 ... yes 3/41 ... yes 13/42 ... yes 18/44 ... yes 3/45 ... yes 3/46 ... yes 16/47 ... yes 2/48 ... yes 16/50 ... yes 18/51 ... yes 2/52 ... yes 20/53 ... yes 13/54 ... yes 23/55 ... yes 23/56 ... yes 23/57 ... yes 23/58 ... yes 3/59 ... yes 3/60 ... yes 26/61 ... yes 3/62 ... yes 13/63 ... yes 3/64 ... yes 3/65 ... yes 2/66 ... yes 29/67 ... yes 23/68 ... yes 3/69 ... yes 29/70 ... yes 20/72 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.26.2) Git user has default SSH configuration? ... yes Active users: ... 9 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished