Gitlab import fails, sidekiq terminating busy threads
Summary
Large repo (3GB with LFS objects) Gitlab import fails, sidekiq appears to terminate busy threads
Steps to reproduce
Attempt to import a largish Gitlab export.
What is the current bug behavior?
After some time (30 mins?), import fails with error message in web UI:
Error importing repository into user/project - tar (child): [REPO EXPORT PATH]/uploads/774b8ad292120fb51165dd8274873e75: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
, Unable to decompress [REPO EXPORT PATH]/uploads/774b8ad292120fb51165dd8274873e75 into [REPO EXPORT PATH]/user/project/47bdd9b6c42711ad4bbe804c138c90d5Attempting to import the exact same Gitlab export will result in same error, just with a different upload hash.
What is the expected correct behavior?
Import succeeds without error.
Relevant logs and/or screenshots
sidekiq.log:
2018-05-17_23:27:18.00258 2018-05-17T23:27:18.002Z 521 TID-ouiu3j9c0 INFO: Booting Sidekiq 5.0.5 with redis options {:host=>"gitlab-gitlab-redis", :port=>6379, :password=>nil, :namespace=>"resque:gitlab", :id=>"Sidekiq-server-PID-521", :url=>nil}
2018-05-17_23:27:18.14629 2018-05-17T23:27:18.146Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: stuck_ci_jobs_worker
2018-05-17_23:27:18.14974 2018-05-17T23:27:18.149Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
2018-05-17_23:27:18.15390 2018-05-17T23:27:18.153Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: pipeline_schedule_worker
2018-05-17_23:27:18.15863 2018-05-17T23:27:18.158Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: repository_check_worker
2018-05-17_23:27:18.16131 2018-05-17T23:27:18.161Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: admin_email_worker
2018-05-17_23:27:18.16482 2018-05-17T23:27:18.164Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: repository_archive_cache_worker
2018-05-17_23:27:18.16704 2018-05-17T23:27:18.166Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: pages_domain_verification_cron_worker
2018-05-17_23:27:18.17018 2018-05-17T23:27:18.170Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: import_export_project_cleanup_worker
2018-05-17_23:27:18.17283 2018-05-17T23:27:18.172Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: requests_profiles_worker
2018-05-17_23:27:18.17533 2018-05-17T23:27:18.175Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: remove_expired_members_worker
2018-05-17_23:27:18.17822 2018-05-17T23:27:18.178Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: remove_expired_group_links_worker
2018-05-17_23:27:18.18049 2018-05-17T23:27:18.180Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: prune_old_events_worker
2018-05-17_23:27:18.18263 2018-05-17T23:27:18.182Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: trending_projects_worker
2018-05-17_23:27:18.18679 2018-05-17T23:27:18.186Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker
2018-05-17_23:27:18.19104 2018-05-17T23:27:18.190Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: stuck_import_jobs_worker
2018-05-17_23:27:18.19463 2018-05-17T23:27:18.194Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: gitlab_usage_ping_worker
2018-05-17_23:27:18.19830 2018-05-17T23:27:18.198Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: schedule_update_user_activity_worker
2018-05-17_23:27:18.20235 2018-05-17T23:27:18.202Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: remove_old_web_hook_logs_worker
2018-05-17_23:27:18.20487 2018-05-17T23:27:18.204Z 521 TID-ouiu3j9c0 INFO: Cron Jobs - add job with name: stuck_merge_jobs_worker
2018-05-17_23:27:33.02752 2018-05-17T23:27:33.027Z 521 TID-ouiu3j9c0 INFO: Running in ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
2018-05-17_23:27:33.02755 2018-05-17T23:27:33.027Z 521 TID-ouiu3j9c0 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2018-05-17_23:27:33.02756 2018-05-17T23:27:33.027Z 521 TID-ouiu3j9c0 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2018-05-17_23:27:33.02923 2018-05-17T23:27:33.029Z 521 TID-ouiu3j9c0 INFO: Starting processing, hit Ctrl-C to stop
2018-05-17_23:27:33.03352 2018-05-17T23:27:33.033Z 521 TID-ouiu3j9c0 INFO: Listening on queues ["authorized_projects", "background_migration", "create_gpg_signature", "cronjob", "cronjob:admin_email", "cronjob:expire_build_artifacts", "cronjob:gitlab_usage_ping", "cronjob:import_export_project_cleanup", "cronjob:pages_domain_verification_cron", "cronjob:pipeline_schedule", "cronjob:prune_old_events", "cronjob:remove_expired_group_links", "cronjob:remove_expired_members", "cronjob:remove_old_web_hook_logs", "cronjob:remove_unreferenced_lfs_objects", "cronjob:repository_archive_cache", "cronjob:repository_check_batch", "cronjob:requests_profiles", "cronjob:schedule_update_user_activity", "cronjob:stuck_ci_jobs", "cronjob:stuck_import_jobs", "cronjob:stuck_merge_jobs", "cronjob:trending_projects", "default", "delete_merged_branches", "delete_user", "email_receiver", "emails_on_push", "expire_build_instance_artifacts", "gcp_cluster", "gcp_cluster:check_gcp_project_billing", "gcp_cluster:cluster_install_app", "gcp_cluster:cluster_provision", "gcp_cluster:cluster_wait_for_app_installation", "gcp_cluster:cluster_wait_for_ingress_ip_address", "gcp_cluster:wait_for_cluster_creation", "git_garbage_collect", "github_import_advance_stage", "github_importer", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", "github_importer:github_import_import_note", "github_importer:github_import_import_pull_request", "github_importer:github_import_refresh_import_jid", "github_importer:github_import_stage_finish_import", "github_importer:github_import_stage_import_base_data", "github_importer:github_import_stage_import_issues_and_diff_notes", "github_importer:github_import_stage_import_notes", "github_importer:github_import_stage_import_pull_requests", "github_importer:github_import_stage_import_repository", "gitlab_shell", "group_destroy", "invalid_gpg_signature_update", "irker", "mailers", "merge", "namespaceless_project_destroy", "new_issue", "new_merge_request", "new_note", "object_storage", "object_storage:object_storage_background_move", "object_storage:object_storage_migrate_uploads", "object_storage_upload", "pages", "pages_domain_verification", "pipeline_background", "pipeline_background:archive_trace", "pipeline_cache", "pipeline_cache:expire_job_cache", "pipeline_cache:expire_pipeline_cache", "pipeline_creation", "pipeline_creation:create_pipeline", "pipeline_creation:run_pipeline_schedule", "pipeline_default", "pipeline_default:build_coverage", "pipeline_default:build_trace_sections", "pipeline_default:pipeline_metrics", "pipeline_default:pipeline_notification", "pipeline_hooks", "pipeline_hooks:build_hooks", "pipeline_hooks:pipeline_hooks", "pipeline_processing", "pipeline_processing:build_finished", "pipeline_processing:build_queue", "pipeline_processing:build_success", "pipeline_processing:pipeline_process", "pipeline_processing:pipeline_success", "pipeline_processing:pipeline_update", "pipeline_processing:stage_update", "pipeline_processing:update_head_pipeline_for_merge_request", "plugin", "post_receive", "process_commit", "project_cache", "project_destroy", "project_export", "project_migrate_hashed_storage", "project_service", "propagate_service_template", "reactive_caching", "rebase", "repository_check", "repository_check:repository_check_clear", "repository_check:repository_check_single_repository", "repository_fork", "repository_import", "storage_migrator", "system_hook_push", "update_merge_requests", "update_user_activity", "upload_checksum", "web_hook"]
2018-05-17_23:29:22.19449 2018-05-17T23:29:22.194Z 521 TID-ouiovnnxo ProjectDestroyWorker JID-980c83aaa9422ade3c9f0547 INFO: start
2018-05-17_23:29:24.00046 2018-05-17T23:29:24.000Z 521 TID-ouiovnnxo ProjectDestroyWorker JID-980c83aaa9422ade3c9f0547 INFO: done: 1.806 sec
2018-05-17_23:30:07.35368 2018-05-17T23:30:07.352Z 521 TID-ouio6clsw PagesDomainVerificationCronWorker JID-9c3a23ea63f0508ceb21f70f INFO: start
2018-05-17_23:30:07.38390 2018-05-17T23:30:07.383Z 521 TID-ouio6clsw PagesDomainVerificationCronWorker JID-9c3a23ea63f0508ceb21f70f INFO: done: 0.031 sec
2018-05-17_23:34:26.32227 2018-05-17T23:34:26.321Z 521 TID-ouiovnn48 GitlabShellWorker JID-fb23ec6b3d1a4daa7797dfc2 INFO: start
2018-05-17_23:34:26.32685 2018-05-17T23:34:26.326Z 521 TID-ouiovnme4 GitlabShellWorker JID-f60e499f30e614f74276a35e INFO: start
2018-05-17_23:34:26.34162 2018-05-17T23:34:26.341Z 521 TID-ouiovnn48 GitlabShellWorker JID-fb23ec6b3d1a4daa7797dfc2 INFO: done: 0.02 sec
2018-05-17_23:34:26.37708 2018-05-17T23:34:26.376Z 521 TID-ouiovnme4 GitlabShellWorker JID-f60e499f30e614f74276a35e INFO: done: 0.05 sec
2018-05-17_23:45:42.38729 2018-05-17T23:45:42.386Z 521 TID-ouiovjlo4 PagesDomainVerificationCronWorker JID-ae898e28bc462e0e08f70eb9 INFO: start
2018-05-17_23:45:42.41058 2018-05-17T23:45:42.410Z 521 TID-ouiovjlo4 PagesDomainVerificationCronWorker JID-ae898e28bc462e0e08f70eb9 INFO: done: 0.024 sec
2018-05-17_23:50:25.45650 2018-05-17T23:50:25.454Z 521 TID-ouio6cgiw ExpireBuildArtifactsWorker JID-e614f063da13f520acf5418e INFO: start
2018-05-17_23:50:25.49995 2018-05-17T23:50:25.499Z 521 TID-ouio6cgiw ExpireBuildArtifactsWorker JID-e614f063da13f520acf5418e INFO: done: 0.045 sec
2018-05-18_00:00:16.66200 2018-05-18T00:00:16.661Z 521 TID-ouiovnoqk StuckCiJobsWorker JID-c9e6c93a4dfb73a444a4d535 INFO: start
2018-05-18_00:00:16.68024 2018-05-18T00:00:16.679Z 521 TID-ouiovno88 RequestsProfilesWorker JID-f7321810932718d7336f2ffb INFO: start
2018-05-18_00:00:16.68680 2018-05-18T00:00:16.686Z 521 TID-ouiovno88 RequestsProfilesWorker JID-f7321810932718d7336f2ffb INFO: done: 0.007 sec
2018-05-18_00:00:16.68802 2018-05-18T00:00:16.687Z 521 TID-ouio6cl8c PagesDomainVerificationCronWorker JID-2c125747c2233c49746abdd1 INFO: start
2018-05-18_00:00:16.69121 2018-05-18T00:00:16.690Z 521 TID-ouiovnoqk StuckCiJobsWorker JID-c9e6c93a4dfb73a444a4d535 INFO: done: 0.029 sec
2018-05-18_00:00:16.69792 2018-05-18T00:00:16.697Z 521 TID-ouiovnnos RepositoryArchiveCacheWorker JID-9c2c9f8c60c95a2de730fe8d INFO: start
2018-05-18_00:00:16.70644 2018-05-18T00:00:16.706Z 521 TID-ouiovnnos RepositoryArchiveCacheWorker JID-9c2c9f8c60c95a2de730fe8d INFO: done: 0.008 sec
2018-05-18_00:00:16.70808 2018-05-18T00:00:16.707Z 521 TID-ouio6cl8c PagesDomainVerificationCronWorker JID-2c125747c2233c49746abdd1 INFO: done: 0.02 sec
2018-05-18_00:00:16.70989 2018-05-18T00:00:16.709Z 521 TID-ouio6cl7s StuckMergeJobsWorker JID-0ca4069e86f8ee7e3e2ba99b INFO: start
2018-05-18_00:00:16.71640 2018-05-18T00:00:16.716Z 521 TID-ouio6clsw ImportExportProjectCleanupWorker JID-c338939296e3a63ceef7f2de INFO: start
2018-05-18_00:00:16.72426 2018-05-18T00:00:16.723Z 521 TID-ouio6ci6w PruneOldEventsWorker JID-5fa598c2be49f4d114e0b946 INFO: start
2018-05-18_00:00:16.73620 2018-05-18T00:00:16.735Z 521 TID-ouio6cl7s StuckMergeJobsWorker JID-0ca4069e86f8ee7e3e2ba99b INFO: done: 0.026 sec
2018-05-18_00:00:16.74656 2018-05-18T00:00:16.746Z 521 TID-ouio6ci6w PruneOldEventsWorker JID-5fa598c2be49f4d114e0b946 INFO: done: 0.023 sec
2018-05-18_00:00:16.80780 2018-05-18T00:00:16.807Z 521 TID-ouio6clsw ImportExportProjectCleanupWorker JID-c338939296e3a63ceef7f2de INFO: done: 0.091 sec
2018-05-18_00:10:03.29029 2018-05-18T00:10:03.290Z 521 TID-ouiovnnos RemoveExpiredGroupLinksWorker JID-593d37613bd99465aeaee897 INFO: start
2018-05-18_00:10:03.30253 2018-05-18T00:10:03.302Z 521 TID-ouiovnoqk RemoveExpiredMembersWorker JID-b5162d4ea9e7a25a53f8990f INFO: start
2018-05-18_00:10:03.33223 2018-05-18T00:10:03.331Z 521 TID-ouiovnnos RemoveExpiredGroupLinksWorker JID-593d37613bd99465aeaee897 INFO: done: 0.041 sec
2018-05-18_00:10:03.33863 2018-05-18T00:10:03.338Z 521 TID-ouiovnoqk RemoveExpiredMembersWorker JID-b5162d4ea9e7a25a53f8990f INFO: done: 0.036 sec
2018-05-18_00:15:12.31278 2018-05-18T00:15:12.312Z 521 TID-ouio6cgiw PagesDomainVerificationCronWorker JID-a5e9de6ca06605e9bbb8ac1a INFO: start
2018-05-18_00:15:12.33327 2018-05-18T00:15:12.332Z 521 TID-ouiovnoyw StuckImportJobsWorker JID-cfcb4ff2d658a3033423ea89 INFO: start
2018-05-18_00:15:12.34425 2018-05-18T00:15:12.344Z 521 TID-ouio6cgiw PagesDomainVerificationCronWorker JID-a5e9de6ca06605e9bbb8ac1a INFO: done: 0.031 sec
2018-05-18_00:15:12.35137 2018-05-18T00:15:12.350Z 521 TID-ouiovnoyw StuckImportJobsWorker JID-cfcb4ff2d658a3033423ea89 INFO: done: 0.018 sec
2018-05-18_00:19:31.59070 2018-05-18T00:19:31.589Z 521 TID-ouio6clm8 PipelineScheduleWorker JID-b42c1e02ec0ed59b3d8451d2 INFO: start
2018-05-18_00:19:31.62145 2018-05-18T00:19:31.621Z 521 TID-ouio6clm8 PipelineScheduleWorker JID-b42c1e02ec0ed59b3d8451d2 INFO: done: 0.032 sec
2018-05-18_00:20:12.47841 2018-05-18T00:20:12.477Z 521 TID-ouiovnmsk RemoveUnreferencedLfsObjectsWorker JID-1fc123d7efc0829bf82b345b INFO: start
2018-05-18_00:20:12.50386 2018-05-18T00:20:12.503Z 521 TID-ouio6cl7s RepositoryCheck::BatchWorker JID-815799975c72f33ff527a444 INFO: start
2018-05-18_00:20:12.51131 2018-05-18T00:20:12.511Z 521 TID-ouiovnmsk RemoveUnreferencedLfsObjectsWorker JID-1fc123d7efc0829bf82b345b INFO: done: 0.034 sec
2018-05-18_00:20:12.52131 2018-05-18T00:20:12.521Z 521 TID-ouio6cl7s RepositoryCheck::BatchWorker JID-815799975c72f33ff527a444 INFO: done: 0.017 sec
2018-05-18_00:30:14.07492 2018-05-18T00:30:14.074Z 521 TID-ouiovnme4 ScheduleUpdateUserActivityWorker JID-669788ff1bb0d7373972d9af INFO: start
2018-05-18_00:30:14.07843 2018-05-18T00:30:14.077Z 521 TID-ouiovnmlc UpdateUserActivityWorker JID-c9fe60d649d0f82c69eb49b6 INFO: start
2018-05-18_00:30:14.09621 2018-05-18T00:30:14.095Z 521 TID-ouiovnme4 ScheduleUpdateUserActivityWorker JID-669788ff1bb0d7373972d9af INFO: done: 0.021 sec
2018-05-18_00:30:14.09817 2018-05-18T00:30:14.098Z 521 TID-ouiovnmsk PagesDomainVerificationCronWorker JID-11b58d80dd71fa1bb1cb8d66 INFO: start
2018-05-18_00:30:14.12637 2018-05-18T00:30:14.126Z 521 TID-ouiovnmsk PagesDomainVerificationCronWorker JID-11b58d80dd71fa1bb1cb8d66 INFO: done: 0.028 sec
2018-05-18_00:30:14.12742 2018-05-18T00:30:14.127Z 521 TID-ouiovnmlc UpdateUserActivityWorker JID-c9fe60d649d0f82c69eb49b6 INFO: done: 0.05 sec
2018-05-18_00:40:03.86376 2018-05-18T00:40:03.863Z 521 TID-ouiovnoyw RemoveOldWebHookLogsWorker JID-a88e714f62d3eb8d03cd6162 INFO: start
2018-05-18_00:40:03.88575 2018-05-18T00:40:03.885Z 521 TID-ouiovnoyw RemoveOldWebHookLogsWorker JID-a88e714f62d3eb8d03cd6162 INFO: done: 0.022 sec
2018-05-18_00:43:39.96083 2018-05-18T00:43:39.959Z 521 TID-ouiovjlo4 RepositoryImportWorker JID-34358f3b2c874d694e2397e8 INFO: start
2018-05-18_00:45:12.48419 2018-05-18T00:45:12.482Z 521 TID-ouio6cl8c PagesDomainVerificationCronWorker JID-af57f8f5409e9c8b69560800 INFO: start
2018-05-18_00:45:12.52041 2018-05-18T00:45:12.520Z 521 TID-ouio6cl8c PagesDomainVerificationCronWorker JID-af57f8f5409e9c8b69560800 INFO: done: 0.038 sec
2018-05-18_00:45:12.52149 2018-05-18T00:45:12.520Z 521 TID-ouig9l82c WARN: Sidekiq worker PID-521 shutting down because of current RSS 1133704 exceeds maximum RSS 1000000 after job PagesDomainVerificationCronWorker JID-af57f8f5409e9c8b69560800
2018-05-18_00:45:12.52168 2018-05-18T00:45:12.521Z 521 TID-ouig9l82c WARN: Sidekiq worker PID-521 will stop fetching new jobs in 900 seconds, and will be shut down 30 seconds later
2018-05-18_00:45:12.52169 2018-05-18T00:45:12.521Z 521 TID-ouig9l82c WARN: waiting 900 seconds before sending Sidekiq worker PID-521 SIGTSTP (stop fetching new jobs)
2018-05-18_00:50:15.21161 2018-05-18T00:50:15.211Z 521 TID-ouiovnme4 ExpireBuildArtifactsWorker JID-867dae6b46b13a4d247dfdb1 INFO: start
2018-05-18_00:50:15.27810 2018-05-18T00:50:15.277Z 521 TID-ouiovnme4 ExpireBuildArtifactsWorker JID-867dae6b46b13a4d247dfdb1 INFO: done: 0.066 sec
2018-05-18_01:00:12.70044 2018-05-18T01:00:12.700Z 521 TID-ouig9l82c WARN: sending Sidekiq worker PID-521 SIGTSTP (stop fetching new jobs)
2018-05-18_01:00:12.70056 2018-05-18T01:00:12.700Z 521 TID-ouig9l82c WARN: waiting 30 seconds before sending Sidekiq worker PID-521 SIGTERM (gracefully shut down)
2018-05-18_01:00:12.78228 2018-05-18T01:00:12.781Z 521 TID-ouiu3j9c0 INFO: Received TSTP, no longer accepting new work
2018-05-18_01:00:12.78231 2018-05-18T01:00:12.781Z 521 TID-ouio6ckjw INFO: Scheduler exiting...
2018-05-18_01:00:12.78302 2018-05-18T01:00:12.782Z 521 TID-ouiu3j9c0 INFO: Terminating quiet workers
2018-05-18_01:00:12.78304 2018-05-18T01:00:12.782Z 521 TID-ouiovjlvw INFO: Scheduler exiting...
2018-05-18_01:00:42.70246 2018-05-18T01:00:42.702Z 521 TID-ouig9l82c WARN: sending Sidekiq worker PID-521 SIGTERM (gracefully shut down)
2018-05-18_01:00:42.70251 2018-05-18T01:00:42.702Z 521 TID-ouig9l82c WARN: waiting 6 seconds before sending Sidekiq worker PID-521 SIGKILL (die)
2018-05-18_01:00:42.80324 2018-05-18T01:00:42.802Z 521 TID-ouiu3j9c0 INFO: Shutting down
2018-05-18_01:00:43.30366 2018-05-18T01:00:43.303Z 521 TID-ouiu3j9c0 INFO: Pausing to allow workers to finish...
2018-05-18_01:00:46.30871 2018-05-18T01:00:46.308Z 521 TID-ouiu3j9c0 WARN: Terminating 1 busy worker threads
2018-05-18_01:00:46.30996 2018-05-18T01:00:46.309Z 521 TID-ouiu3j9c0 WARN: Work still in progress [#<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:repository_import", job="{\"retry\":false,\"args\":[35],\"class\":\"RepositoryImportWorker\",\"queue\":\"repository_import\",\"backtrace\":5,\"status_expiration\":54000,\"jid\":\"34358f3b2c874d694e2397e8\",\"created_at\":1526604219.9575596,\"enqueued_at\":1526604219.9584177}">]
2018-05-18_01:00:46.31077 2018-05-18T01:00:46.310Z 521 TID-ouiu3j9c0 INFO: Pushed 1 jobs back to Redis
2018-05-18_01:00:46.31182 2018-05-18T01:00:46.311Z 521 TID-ouiu3j9c0 INFO: Bye!
2018-05-18_01:01:16.25432 2018-05-18T01:01:16.254Z 12031 TID-ouuscaneo INFO: Booting Sidekiq 5.0.5 with redis options {:host=>"gitlab-gitlab-redis", :port=>6379, :password=>nil, :namespace=>"resque:gitlab", :id=>"Sidekiq-server-PID-12031", :url=>nil}
2018-05-18_01:01:16.34813 2018-05-18T01:01:16.348Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: stuck_ci_jobs_worker
2018-05-18_01:01:16.35094 2018-05-18T01:01:16.350Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
2018-05-18_01:01:16.35317 2018-05-18T01:01:16.352Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: pipeline_schedule_worker
2018-05-18_01:01:16.35520 2018-05-18T01:01:16.355Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: repository_check_worker
2018-05-18_01:01:16.35691 2018-05-18T01:01:16.356Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: admin_email_worker
2018-05-18_01:01:16.35969 2018-05-18T01:01:16.359Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: repository_archive_cache_worker
2018-05-18_01:01:16.36156 2018-05-18T01:01:16.361Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: pages_domain_verification_cron_worker
2018-05-18_01:01:16.36456 2018-05-18T01:01:16.364Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: import_export_project_cleanup_worker
2018-05-18_01:01:16.36663 2018-05-18T01:01:16.366Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: requests_profiles_worker
2018-05-18_01:01:16.36899 2018-05-18T01:01:16.368Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: remove_expired_members_worker
2018-05-18_01:01:16.37136 2018-05-18T01:01:16.371Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: remove_expired_group_links_worker
2018-05-18_01:01:16.37296 2018-05-18T01:01:16.372Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: prune_old_events_worker
2018-05-18_01:01:16.37635 2018-05-18T01:01:16.376Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: trending_projects_worker
2018-05-18_01:01:16.37886 2018-05-18T01:01:16.378Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker
2018-05-18_01:01:16.38060 2018-05-18T01:01:16.380Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: stuck_import_jobs_worker
2018-05-18_01:01:16.38302 2018-05-18T01:01:16.382Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: gitlab_usage_ping_worker
2018-05-18_01:01:16.38583 2018-05-18T01:01:16.385Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: schedule_update_user_activity_worker
2018-05-18_01:01:16.38892 2018-05-18T01:01:16.388Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: remove_old_web_hook_logs_worker
2018-05-18_01:01:16.39065 2018-05-18T01:01:16.390Z 12031 TID-ouuscaneo INFO: Cron Jobs - add job with name: stuck_merge_jobs_worker
2018-05-18_01:01:28.64023 2018-05-18T01:01:28.640Z 12031 TID-ouuscaneo INFO: Running in ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
2018-05-18_01:01:28.64029 2018-05-18T01:01:28.640Z 12031 TID-ouuscaneo INFO: See LICENSE and the LGPL-3.0 for licensing details.
2018-05-18_01:01:28.64030 2018-05-18T01:01:28.640Z 12031 TID-ouuscaneo INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2018-05-18_01:01:28.64184 2018-05-18T01:01:28.641Z 12031 TID-ouuscaneo INFO: Starting processing, hit Ctrl-C to stop
2018-05-18_01:01:28.64586 2018-05-18T01:01:28.645Z 12031 TID-ouuscaneo INFO: Listening on queues ["authorized_projects", "background_migration", "create_gpg_signature", "cronjob", "cronjob:admin_email", "cronjob:expire_build_artifacts", "cronjob:gitlab_usage_ping", "cronjob:import_export_project_cleanup", "cronjob:pages_domain_verification_cron", "cronjob:pipeline_schedule", "cronjob:prune_old_events", "cronjob:remove_expired_group_links", "cronjob:remove_expired_members", "cronjob:remove_old_web_hook_logs", "cronjob:remove_unreferenced_lfs_objects", "cronjob:repository_archive_cache", "cronjob:repository_check_batch", "cronjob:requests_profiles", "cronjob:schedule_update_user_activity", "cronjob:stuck_ci_jobs", "cronjob:stuck_import_jobs", "cronjob:stuck_merge_jobs", "cronjob:trending_projects", "default", "delete_merged_branches", "delete_user", "email_receiver", "emails_on_push", "expire_build_instance_artifacts", "gcp_cluster", "gcp_cluster:check_gcp_project_billing", "gcp_cluster:cluster_install_app", "gcp_cluster:cluster_provision", "gcp_cluster:cluster_wait_for_app_installation", "gcp_cluster:cluster_wait_for_ingress_ip_address", "gcp_cluster:wait_for_cluster_creation", "git_garbage_collect", "github_import_advance_stage", "github_importer", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", "github_importer:github_import_import_note", "github_importer:github_import_import_pull_request", "github_importer:github_import_refresh_import_jid", "github_importer:github_import_stage_finish_import", "github_importer:github_import_stage_import_base_data", "github_importer:github_import_stage_import_issues_and_diff_notes", "github_importer:github_import_stage_import_notes", "github_importer:github_import_stage_import_pull_requests", "github_importer:github_import_stage_import_repository", "gitlab_shell", "group_destroy", "invalid_gpg_signature_update", "irker", "mailers", "merge", "namespaceless_project_destroy", "new_issue", "new_merge_request", "new_note", "object_storage", "object_storage:object_storage_background_move", "object_storage:object_storage_migrate_uploads", "object_storage_upload", "pages", "pages_domain_verification", "pipeline_background", "pipeline_background:archive_trace", "pipeline_cache", "pipeline_cache:expire_job_cache", "pipeline_cache:expire_pipeline_cache", "pipeline_creation", "pipeline_creation:create_pipeline", "pipeline_creation:run_pipeline_schedule", "pipeline_default", "pipeline_default:build_coverage", "pipeline_default:build_trace_sections", "pipeline_default:pipeline_metrics", "pipeline_default:pipeline_notification", "pipeline_hooks", "pipeline_hooks:build_hooks", "pipeline_hooks:pipeline_hooks", "pipeline_processing", "pipeline_processing:build_finished", "pipeline_processing:build_queue", "pipeline_processing:build_success", "pipeline_processing:pipeline_process", "pipeline_processing:pipeline_success", "pipeline_processing:pipeline_update", "pipeline_processing:stage_update", "pipeline_processing:update_head_pipeline_for_merge_request", "plugin", "post_receive", "process_commit", "project_cache", "project_destroy", "project_export", "project_migrate_hashed_storage", "project_service", "propagate_service_template", "reactive_caching", "rebase", "repository_check", "repository_check:repository_check_clear", "repository_check:repository_check_single_repository", "repository_fork", "repository_import", "storage_migrator", "system_hook_push", "update_merge_requests", "update_user_activity", "upload_checksum", "web_hook"]
2018-05-18_01:01:28.76193 2018-05-18T01:01:28.761Z 12031 TID-ouup29898 RepositoryImportWorker JID-34358f3b2c874d694e2397e8 INFO: startIt appears sidekiq terminates jobs due to lack of RAM, but machine has 14.7GB of RAM, should that happen?
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.7.3 Revision: 2555d6c Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.company.com HTTP Clone URL: https://gitlab.company.com/some-group/some-project.git SSH Clone URL: git@gitlab.company.com:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 7.1.2 Repository storage paths:
- default: /gitlab-data/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 7.1.2 ? ... OK (7.1.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 8/4 ... ok 8/11 ... ok 8/12 ... ok 8/15 ... ok 8/18 ... ok 8/21 ... ok 8/23 ... ok 8/27 ... ok 7/35 ... repository is empty Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /gitlab-data/ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 8/4 ... yes 8/11 ... yes 8/12 ... yes 8/15 ... yes 8/18 ... yes 8/21 ... yes 8/23 ... yes 8/27 ... yes 7/35 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 4
Checking GitLab ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
Edited by Mark Nordine