GitLab Docker Image 16.6.1 takes years to start because of "Cleaning stale PIDs & sockets"
Summary
Hi! When upgrading my docker omnibus image (running on kubernetes cluster), the application starts but takes a very long time to process the Cleaning stale PIDS.
My guess is because de /var/opt/gitlab
path was added from !7215 (merged), and I have all my data under it (/var/opt/gitlab/gitlab-rails
), with non-local volumes.
Steps to reproduce
- Have an installation with remote disks for GitLab data (
/var/opt/gitlab/gitlab-rails
) - Start docker image 16.6.1
- the wrapper command
find /var/opt/gitlab ( -type f -o -type s ) ( -name pid -o -name *.pid -o -name socket.? ) -delete
takes years to process.
What is the current bug behavior?
Application does not start in an acceptable time (for my last try, I'm still waiting after 45min).
What is the expected correct behavior?
Application should start in an acceptable time.
Relevant logs
Relevant logs
root@gitlab-0:/# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 4780 3452 ? Ss 09:50 0:00 /bin/bash /assets/wrapper root 10 9.2 0.0 4988 3320 ? D 09:50 3:19 find /var/opt/gitlab ( -type f -o -type s ) ( -name pid -o -name *.pid -o -name socket.? ) -delete root 11 0.0 0.0 2888 996 pts/0 Ss 09:50 0:00 sh -c clear; (bash || ash || sh) root 18 0.0 0.0 2888 104 pts/0 S 09:50 0:00 sh -c clear; (bash || ash || sh) root 19 0.0 0.0 5044 3840 pts/0 S 09:50 0:00 bash root 32 0.0 0.0 7480 3164 pts/0 R+ 10:26 0:00 ps aux
Details of package version
Provide the package version installation details
Containers: gitlab: Image: gitlab/gitlab-ce:16.6.1-ce.0 Image ID: docker.io/gitlab/gitlab-ce@sha256:c8f60686f4b2703874230d5b02bd10a648bada815e1138e9be0834d0e6307118 Requests: cpu: 1 memory: 12Gi Volumes: etc: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: gitlab-etc ReadOnly: false log: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: gitlab-log ReadOnly: false data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: gitlab-data ReadOnly: false shared: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: gitlab-shared-data ReadOnly: false lfs: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: gitlab-lfs ReadOnly: false backup: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: gitlab-backup ReadOnly: false prometheus: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: gitlab-prometheus ReadOnly: false
Environment details
- Operating System:
Docker Image on Ubuntu 22.04 LTS nodes
- Installation Target, remove incorrect values:
- Other:
Managed Kubernetes Service
with PVC (remote storage)
- Other:
- Installation Type, remove incorrect values:
- Upgrade from version
16.5.3
- Upgrade from version
- Is there any other software running on the machine:
Yes
(Kubernetes Cluster) - Is this a single or multiple node installation?
Single
- Resources
- CPU:
1
- Memory total:
12 GB
- CPU:
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://gitlab.opsrel.io/' nginx['listen_port'] = 80 nginx['listen_https'] = false gitlab_rails['gitlab_ssh_host'] = 'gitlab.opsrel.io' gitlab_rails['lfs_enabled'] = true gitlab_rails['omniauth_enabled'] = true gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] gitlab_rails['omniauth_block_auto_created_users'] = false gitlab_rails['omniauth_auto_link_saml_user'] = true gitlab_rails['omniauth_providers'] = [ { name: 'saml', label: 'Keycloak', args: { assertion_consumer_service_url: 'https://gitlab.opsrel.io/users/auth/saml/callback', idp_cert: "xxxx", idp_sso_target_url: 'https://keycloak.opsrel.io/auth/realms/software-factory/protocol/saml', issuer: 'https://gitlab.opsrel.io', name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', attribute_statements: { username: ['username'] } } } ] gitlab_rails['gitlab_shell_ssh_port'] = 30000 gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = "mailcatcher.smtp" gitlab_rails['smtp_port'] = 25 gitlab_rails['gitlab_email_from'] = 'gitlab@opsrel.io' gitlab_rails['gitlab_email_reply_to'] = 'gitlab@opsrel.io' gitlab_rails['gitlab_email_display_name'] = 'Gitlab' gitlab_rails['smtp_enable_starttls_auto'] = true gitlab_rails['monitoring_whitelist'] = ['0.0.0.0/0'] puma['worker_timeout'] = 1200 puma['worker_processes'] = 8 nginx['client_max_body_size'] = '15000m'
Edited by Rémi Verchère