Sidekiq in crashloop 2.20
Summary
Have upgrade to chart 2.2.0 and seeing constant restarts of sidekiq pod. Logs are not outputting any error messages.
Steps to reproduce
Upgrade from 2.0.3 => 2.2.0
Configuration used
global:
hosts:
domain: example.com
https: true
registry:
https: true
ingress:
configureCertmanager: false
enabled: true
psql:
password:
secret: gitlab-psql-pw
key: postgres-password
host: db
username: gitlab
database: gitlab
redis:
password:
key: key
secret: gitlab-redis-pw
host: redis-redis-master-svc.redis
minio:
enabled: false
appConfig:
lfs:
bucket: ap-gitlab-lfs
connection:
secret: gitlab-gcloud-bucket-sa
key: connection
artifacts:
bucket: ap-gitlab-artifacts
connection:
secret: gitlab-gcloud-bucket-sa
key: connection
uploads:
enabled: true
bucket: ap-gitlab-uploads
connection:
secret: gitlab-gcloud-bucket-sa
key: connection
packages:
enabled: false
backups:
bucket: ap-gitlab-backup
tmpBucket: ap-gitlab-backup-tmp
omniauth:
enabled: true
autoSignInWithProvider:
syncProfileFromProvider: []
syncProfileAttributes: ['email']
allowSingleSignOn: ['saml']
blockAutoCreatedUsers: true
autoLinkLdapUser: false
autoLinkSamlUser: true
externalProviders: []
providers:
- secret: gitlab-okta-saml
registry:
bucket: ap-gitlab-registry
smtp:
enabled: true
address: redacted
port: 25
user_name: ""
password:
secret: ""
key: password
# domain:
authentication: false
starttls_auto: true
openssl_verify_mode: "none"
email:
from: redacted
display_name: GitLab
reply_to: redacted
subject_suffix: ''
time_zone: UTC
postgresql:
install: false
prometheus:
install: false
gitlab-runner:
install: false
certmanager:
install: false
gitlab:
nginx-ingress:
enabled: true
#tcp:
# 22: "enabled"
tcpExternalConfig: "true"
controller:
config:
hsts-include-subdomains: "false"
server-name-hash-bucket-size: "256"
enable-vts-status: "true"
use-http2: "true"
ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"
ssl-protocols: "TLSv1.1 TLSv1.2"
server-tokens: "false"
extraArgs:
force-namespace-isolation: "true"
service:
annotations:
cloud.google.com/load-balancer-type: Internal
externalTrafficPolicy: "Local"
resources:
requests:
cpu: 50m
memory: 100Mi
limits:
cpu: 300m
memory: 200Mi
publishService:
enabled: true
replicaCount: 1
minAvailable: 0
scope:
enabled: true
stats:
enabled: true
metrics:
enabled: true
service:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "10254"
defaultBackend:
minAvailable: 1
replicaCount: 1
resources:
requests:
cpu: 50m
memory: 100Mi
limits:
cpu: 300m
memory: 500Mi
rbac:
create: true
createClusterRole: true
createRole: true
redis-ha:
enabled: false
registry:
enabled: true
minReplicas: 1
maxReplicas: 5
ingress:
annotations:
certmanager.k8s.io/cluster-issuer: acme-shared
tls:
secretName: gitlab-registry-tls-secret
init:
resources:
requests:
cpu: 50m
limits:
cpu: 200m
memory: 1Gi
resources:
requests:
cpu: 200m
memory: 200Mi
limits:
cpu: 300m
memory: 1Gi
storage:
secret: gitlab-gcloud-registry-bucket-sa
key: config
extraKey: "gcs-key.json"
redis:
enabled: false
shared-secrets:
enabled: true
rbac:
create: true
gitlab:
task-runner:
backups:
cron:
enabled: true
schedule: "0 2 * * *"
objectStorage:
backend: gcs
config:
gcpProject: redacted
secret: gitlab-gcloud-backup-bucket-sa
key: gcs-key.json
resources:
requests:
cpu: 50m
memory: 350Mi
limits:
cpu: 200m
memory: 1Gi
init:
resources:
requests:
cpu: 50m
memory: 5Mi
limits:
cpu: 200m
memory: 1Gi
resources:
requests:
cpu: 50m
memory: 350Mi
limits:
cpu: 100m
memory: 500Mi
migrations:
enabled: true
unicorn:
enabled: true
replicaCount: 1
maxReplicas: 3
minReplicas: 1
init:
resources:
requests:
cpu: 100m
memory: 768Mi
limits:
cpu: 500m
memory: 1Gi
ingress:
annotations:
certmanager.k8s.io/cluster-issuer: acme-shared
tls:
secretName: gitlab-unicorn-tls-secret
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 500m
memory: 1.5Gi
workhorse:
resources:
requests:
cpu: 250m
memory: 200Mi
limits:
cpu: 500m
memory: 1Gi
sidekiq:
enabled: true
maxReplicas: 3
hpa:
targetAverageValue: 800m
init:
resources:
requests:
cpu: 50m
memory: 5Mi
limits:
cpu: 300m
memory: 1Gi
resources:
requests:
cpu: 250m
memory: 100Mi
limits:
cpu: 1
memory: 250Mi
gitaly:
enabled: true
init:
resources:
requests:
cpu: 5m
memory: 5Mi
limits:
cpu: 200m
memory: 1Gi
resources:
requests:
cpu: 500m
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
persistence:
size: 80Gi
#storageClass: retain-ssd
#volumeName: pvc-276ec32f-cac1-11e8-ae02-42010af0fc18
gitlab-shell:
enabled: true
minReplicas: 1
maxReplicas: 5
hpa:
targetAverageValue: 400m
init:
resources:
requests:
cpu: 50m
memory: 5Mi
limits:
cpu: 300m
memory: 1Gi
resources:
requests:
cpu: 500m
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
Current behavior
Sidekiq process seems to die after a few minutes making the kube liveness probe kick and restart.
Expected behavior
No restarts. Stable sidekiq
Versions
- Chart: 2.2.0
- Platform:
- Cloud: GKE
- Kubernetes:
- Client: 1.13.6
- Server: 1.13.6
- Helm:
- Client: 2.12
- Server: 2.12
Relevant logs
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/sidekiq_queues.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/resque.yml
Copying other config files found in /var/opt/gitlab/templates
Copying smtp_settings.rb into /srv/gitlab/config
+ exec /bin/sh -c /scripts/process-wrapper
Starting Sidekiq
==> /var/log/gitlab/api_json.log <==
# Logfile created on 2019-08-21 21:56:18 +0000 by logger.rb/66358
==> /var/log/gitlab/grpc.log <==
# Logfile created on 2019-08-21 21:56:18 +0000 by logger.rb/66358
==> /var/log/gitlab/production.log <==
Creating scope :join_project. Overwriting existing method MergeRequest.join_project.
Creating scope :references_project. Overwriting existing method MergeRequest.references_project.
Creating scope :with_api_entity_associations. Overwriting existing method MergeRequest.with_api_entity_associations.
Creating scope :system. Overwriting existing method Note.system.
Creating scope :group_view_details. Overwriting existing method User.group_view_details.
Creating scope :support_bot. Overwriting existing method User.support_bot.
Creating scope :alert_bot. Overwriting existing method User.alert_bot.
Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses.
Creating scope :opened. Overwriting existing method Epic.opened.
Creating scope :closed. Overwriting existing method Epic.closed.
==> /var/log/gitlab/production_json.log <==
# Logfile created on 2019-08-21 21:56:18 +0000 by logger.rb/66358
==> /var/log/gitlab/repocheck.log <==
WARNING: This version of GitLab depends on gitlab-shell 9.3.0, but you're running Unknown. Please update gitlab-shell.
==> /var/log/gitlab/production.log <==
Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects.
Creating scope :join_project. Overwriting existing method MergeRequest.join_project.
Creating scope :references_project. Overwriting existing method MergeRequest.references_project.
Creating scope :with_api_entity_associations. Overwriting existing method MergeRequest.with_api_entity_associations.
Creating scope :system. Overwriting existing method Note.system.
Creating scope :group_view_details. Overwriting existing method User.group_view_details.
Creating scope :support_bot. Overwriting existing method User.support_bot.
Creating scope :alert_bot. Overwriting existing method User.alert_bot.
Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses.
Creating scope :opened. Overwriting existing method Epic.opened.
Creating scope :closed. Overwriting existing method Epic.closed.
2019-08-23T12:42:00.234Z 8 TID-gs542tml0 INFO: Booting Sidekiq 5.2.7 with redis options {:scheme=>"redis", :host=>"redis-redis-master-svc.redis", :port=>6379, :password=>"REDACTED", :id=>nil, :namespace=>"resque:gitlab", :url=>nil}
2019-08-23T12:42:00.245Z 8 TID-gs542tml0 INFO: GitLab reliable fetch activated!
2019-08-23T12:42:00.311Z 8 TID-gs542tml0 INFO: Cron Jobs - deleted job with name: update_all_mirrors_worker
2019-08-23T12:42:00.314Z 8 TID-gs542tml0 INFO: Cron Jobs - deleted job with name: geo_sidekiq_cron_config_worker
2019-08-23T12:42:00.318Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: stuck_ci_jobs_worker
2019-08-23T12:42:00.322Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: pipeline_schedule_worker
2019-08-23T12:42:00.326Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
2019-08-23T12:42:00.330Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: repository_check_worker
2019-08-23T12:42:00.334Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: admin_email_worker
2019-08-23T12:42:00.338Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: repository_archive_cache_worker
2019-08-23T12:42:00.342Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: import_export_project_cleanup_worker
2019-08-23T12:42:00.346Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: ci_archive_traces_cron_worker
2019-08-23T12:42:00.349Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: requests_profiles_worker
2019-08-23T12:42:00.354Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: remove_expired_members_worker
2019-08-23T12:42:00.357Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: remove_expired_group_links_worker
2019-08-23T12:42:00.361Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: prune_old_events_worker
2019-08-23T12:42:00.370Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: trending_projects_worker
2019-08-23T12:42:00.383Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker
2019-08-23T12:42:00.387Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: stuck_import_jobs_worker
2019-08-23T12:42:00.390Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: gitlab_usage_ping_worker
2019-08-23T12:42:00.394Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: stuck_merge_jobs_worker
2019-08-23T12:42:00.399Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: pages_domain_verification_cron_worker
2019-08-23T12:42:00.403Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: pages_domain_removal_cron_worker
2019-08-23T12:42:00.405Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: pages_domain_ssl_renewal_cron_worker
2019-08-23T12:42:00.408Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: issue_due_scheduler_worker
2019-08-23T12:42:00.411Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: prune_web_hook_logs_worker
2019-08-23T12:42:00.414Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: schedule_migrate_external_diffs_worker
2019-08-23T12:42:00.417Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: namespaces_prune_aggregation_schedules_worker
2019-08-23T12:42:00.420Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: clear_shared_runners_minutes_worker
2019-08-23T12:42:00.425Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_file_download_dispatch_worker
2019-08-23T12:42:00.428Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_metrics_update_worker
2019-08-23T12:42:00.431Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_migrated_local_files_clean_up_worker
2019-08-23T12:42:00.434Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_prune_event_log_worker
2019-08-23T12:42:00.436Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_repository_sync_worker
2019-08-23T12:42:00.439Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_repository_verification_primary_batch_worker
2019-08-23T12:42:00.441Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_repository_verification_secondary_scheduler_worker
2019-08-23T12:42:00.444Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_container_repository_sync_worker
2019-08-23T12:42:00.447Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: historical_data_worker
2019-08-23T12:42:00.450Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: ldap_group_sync_worker
2019-08-23T12:42:00.453Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: ldap_sync_worker
2019-08-23T12:42:00.455Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: pseudonymizer_worker
2019-08-23T12:42:00.458Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: update_max_seats_used_for_gitlab_com_subscriptions_worker
2019-08-23T12:42:00.555Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: update_all_mirrors_worker
2019-08-23T12:42:00.561Z 8 TID-gs542tml0 INFO: Cron Jobs - add job with name: geo_sidekiq_cron_config_worker
Cheers for the help
Edited by Thomas Miller