Sidekiq fails to start with message: can't create Thread : Resource temporarily unavailable
Summary
After upgrading from v9 to v10, Sidekiq fails to start reporting: can't create Thread : Resource temporarily unavailable
Steps to reproduce
I just try to start it with: sudo -u git -H RAILS_ENV=production bin/background_jobs start
What is the current bug behavior?
Sidekiq fails to start
What is the expected correct behavior?
Sidekiq succeed to start ;)
Relevant logs and/or screenshots
Process doesn't exist
2018-05-26T17:41:45.524Z 1708 TID--9sci7o INFO: Booting Sidekiq 5.0.5 with redis options {:path=>"/var/run/redis/redis.sock", :namespace=>"resque:gitlab", :id=>"Sidekiq-server-PID-1708", :url=>nil}
2018-05-26T17:41:45.704Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: stuck_ci_jobs_worker
2018-05-26T17:41:45.707Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: pipeline_schedule_worker
2018-05-26T17:41:45.707Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
2018-05-26T17:41:45.710Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: repository_check_worker
2018-05-26T17:41:45.711Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: admin_email_worker
2018-05-26T17:41:45.713Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: repository_archive_cache_worker
2018-05-26T17:41:45.715Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: pages_domain_verification_cron_worker
2018-05-26T17:41:45.716Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: import_export_project_cleanup_worker
2018-05-26T17:41:45.717Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: requests_profiles_worker
2018-05-26T17:41:45.719Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: remove_expired_members_worker
2018-05-26T17:41:45.721Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: remove_expired_group_links_worker
2018-05-26T17:41:45.722Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: prune_old_events_worker
2018-05-26T17:41:45.723Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: trending_projects_worker
2018-05-26T17:41:45.726Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker
2018-05-26T17:41:45.728Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: stuck_import_jobs_worker
2018-05-26T17:41:45.730Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: gitlab_usage_ping_worker
2018-05-26T17:41:45.733Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: schedule_update_user_activity_worker
2018-05-26T17:41:45.735Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: remove_old_web_hook_logs_worker
2018-05-26T17:41:45.736Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: stuck_merge_jobs_worker
2018-05-26T17:41:45.774Z 1708 TID--9sci7o INFO: Cron Jobs - add job with name: issue_due_scheduler_worker
2018-05-26T17:42:12.382Z 1708 TID--9sci7o INFO: Running in ruby 2.3.7p456 (2018-03-28 revision 63024) [i686-linux]
2018-05-26T17:42:12.383Z 1708 TID--9sci7o INFO: See LICENSE and the LGPL-3.0 for licensing details.
2018-05-26T17:42:12.383Z 1708 TID--9sci7o INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2018-05-26T17:42:12.388Z 1708 TID--9sci7o INFO: Listening on queues ["authorized_projects", "background_migration", "create_gpg_signature", "cronjob", "cronjob:admin_email", "cronjob:expire_build_artifacts", "cronjob:gitlab_usage_ping", "cronjob:import_export_project_cleanup", "cronjob:issue_due_scheduler", "cronjob:pages_domain_verification_cron", "cronjob:pipeline_schedule", "cronjob:prune_old_events", "cronjob:remove_expired_group_links", "cronjob:remove_expired_members", "cronjob:remove_old_web_hook_logs", "cronjob:remove_unreferenced_lfs_objects", "cronjob:repository_archive_cache", "cronjob:repository_check_batch", "cronjob:requests_profiles", "cronjob:schedule_update_user_activity", "cronjob:stuck_ci_jobs", "cronjob:stuck_import_jobs", "cronjob:stuck_merge_jobs", "cronjob:trending_projects", "default", "delete_merged_branches", "delete_user", "email_receiver", "emails_on_push", "expire_build_instance_artifacts", "gcp_cluster", "gcp_cluster:check_gcp_project_billing", "gcp_cluster:cluster_install_app", "gcp_cluster:cluster_provision", "gcp_cluster:cluster_wait_for_app_installation", "gcp_cluster:cluster_wait_for_ingress_ip_address", "gcp_cluster:wait_for_cluster_creation", "git_garbage_collect", "github_import_advance_stage", "github_importer", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", "github_importer:github_import_import_note", "github_importer:github_import_import_pull_request", "github_importer:github_import_refresh_import_jid", "github_importer:github_import_stage_finish_import", "github_importer:github_import_stage_import_base_data", "github_importer:github_import_stage_import_issues_and_diff_notes", "github_importer:github_import_stage_import_notes", "github_importer:github_import_stage_import_pull_requests", "github_importer:github_import_stage_import_repository", "gitlab_shell", "group_destroy", "invalid_gpg_signature_update", "irker", "mail_scheduler", "mail_scheduler:mail_scheduler_issue_due", "mail_scheduler:mail_scheduler_notification_service", "mailers", "merge", "namespaceless_project_destroy", "new_issue", "new_merge_request", "new_note", "object_storage", "object_storage:object_storage_background_move", "object_storage:object_storage_migrate_uploads", "object_storage_upload", "pages", "pages_domain_verification", "pipeline_background", "pipeline_background:archive_trace", "pipeline_background:ci_build_trace_chunk_flush", "pipeline_cache", "pipeline_cache:expire_job_cache", "pipeline_cache:expire_pipeline_cache", "pipeline_creation", "pipeline_creation:create_pipeline", "pipeline_creation:run_pipeline_schedule", "pipeline_default", "pipeline_default:build_coverage", "pipeline_default:build_trace_sections", "pipeline_default:pipeline_metrics", "pipeline_default:pipeline_notification", "pipeline_hooks", "pipeline_hooks:build_hooks", "pipeline_hooks:pipeline_hooks", "pipeline_processing", "pipeline_processing:build_finished", "pipeline_processing:build_queue", "pipeline_processing:build_success", "pipeline_processing:pipeline_process", "pipeline_processing:pipeline_success", "pipeline_processing:pipeline_update", "pipeline_processing:stage_update", "pipeline_processing:update_head_pipeline_for_merge_request", "plugin", "post_receive", "process_commit", "project_cache", "project_destroy", "project_export", "project_migrate_hashed_storage", "project_service", "propagate_service_template", "reactive_caching", "rebase", "repository_check", "repository_check:repository_check_clear", "repository_check:repository_check_single_repository", "repository_fork", "repository_import", "repository_remove_remote", "repository_update_remote_mirror", "storage_migrator", "system_hook_push", "update_merge_requests", "update_user_activity", "upload_checksum", "web_hook"]
can't create Thread: Resource temporarily unavailable
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:23:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:23:in `new'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:23:in `safe_thread'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:65:in `start'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/manager.rb:47:in `block in start'
/usr/local/lib/ruby/2.3.0/set.rb:306:in `each_key'
/usr/local/lib/ruby/2.3.0/set.rb:306:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/manager.rb:46:in `start'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/launcher.rb:27:in `run'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-cron-0.6.0/lib/sidekiq/cron/launcher.rb:31:in `run'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/cli.rb:105:in `run'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/bin/sidekiq:12:in `<top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/bin/sidekiq:23:in `load'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/bin/sidekiq:23:in `<top (required)>'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/cli/exec.rb:74:in `load'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/cli/exec.rb:74:in `kernel_load'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/cli/exec.rb:28:in `run'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/cli.rb:424:in `exec'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/cli.rb:27:in `dispatch'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/cli.rb:18:in `start'
/usr/local/lib/ruby/gems/2.3.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>'
/usr/local/lib/ruby/site_ruby/2.3.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.3.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Output of checks
This bug happens on a private Gitlab CE
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 9.4 Current User: git Using RVM: no Ruby Version: 2.3.7p456 Gem Version: 2.7.7 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.2.8 Git Version: 2.16.3 Sidekiq Version:5.0.5 Go Version: go1.10.2 linux/386
GitLab information Version: 10.8.1 Revision: 21a8d61d5ca Directory: /home/git/gitlab DB Adapter: postgresql URL: http://gitlab.wadane.com:81 HTTP Clone URL: http://gitlab.wadane.com:81/some-group/some-project.git SSH Clone URL: git@gitlab.wadane.com:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 7.1.2 Repository storage paths:
- default: /home/git/repositories Hooks: /home/git/gitlab-shell/hooks Git: /usr/local/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 7.1.2 ? ... OK (7.1.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 2/2 ... ok 9/3 ... ok 9/4 ... ok 9/5 ... ok 9/6 ... ok 9/7 ... ok 9/8 ... ok 9/9 ... ok 9/10 ... ok 9/11 ... ok 9/12 ... ok 9/13 ... ok 9/14 ... ok 9/15 ... ok 9/16 ... ok Running /home/git/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /home/git/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... yes Init script up-to-date? ... yes Projects have namespace: ... 2/2 ... yes 9/3 ... yes 9/4 ... yes 9/5 ... yes 9/6 ... yes 9/7 ... yes 9/8 ... yes 9/9 ... yes 9/10 ... yes 9/11 ... yes 9/12 ... yes 9/13 ... yes 9/14 ... yes 9/15 ... yes 9/16 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.7) Git version >= 2.9.5 ? ... yes (2.16.3) Git user has default SSH configuration? ... yes Active users: ... 5
Checking GitLab ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)