Github importer stuck in progress when failing to import repository
Summary
When trying to import rails/rails repository from Github, ImportRepositoryWorker will fail, but import will be stuck in progress indefinitely.
Steps to reproduce
Try to import rails/rails project from Github.
What is the current bug behavior?
Import stuck as in progress indefinitely.
What is the expected correct behavior?
The exception should be registered in ImportFailures and the ImportState should be updated to failed after exceptions that block the import.
Relevant logs and/or screenshots
importer.log
{"severity":"INFO","time":"2021-07-08T12:44:21.141Z","correlation_id":"01FA32BGXJKF46C2269KB1JCKF","message":"starting stage","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:44:21.154Z","correlation_id":"01FA32BGXJKF46C2269KB1JCKF","message":"starting importer","importer":"Importer::RepositoryImporter","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"ERROR","time":"2021-07-08T12:44:30.036Z","correlation_id":"01FA32BGXJKF46C2269KB1JCKF","message":"stage failed","error.message":"2:fetch remote: \"error: object 3b1316b37e444cb7378f09193f556ebee2541cb7: zeroPaddedFilemode: contains zero-padded file modes\\nfatal: fsck error in packed object\\nfatal: fetch-pack: invalid index-pack output\\n\": exit status 128.","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:44:52.620Z","correlation_id":"12088d3c4cf03dd9db4f42ae47ac8a99","message":"starting stage","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:44:52.630Z","correlation_id":"12088d3c4cf03dd9db4f42ae47ac8a99","message":"starting importer","importer":"Importer::RepositoryImporter","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"ERROR","time":"2021-07-08T12:45:00.356Z","correlation_id":"12088d3c4cf03dd9db4f42ae47ac8a99","message":"stage failed","error.message":"2:fetch remote: \"error: object 3b1316b37e444cb7378f09193f556ebee2541cb7: zeroPaddedFilemode: contains zero-padded file modes\\nfatal: fsck error in packed object\\nfatal: fetch-pack: invalid index-pack output\\n\": exit status 128.","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:45:45.655Z","correlation_id":"57903cb93b9f7056662e31c8d2c769ea","message":"starting stage","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:45:45.665Z","correlation_id":"57903cb93b9f7056662e31c8d2c769ea","message":"starting importer","importer":"Importer::RepositoryImporter","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"ERROR","time":"2021-07-08T12:45:54.108Z","correlation_id":"57903cb93b9f7056662e31c8d2c769ea","message":"stage failed","error.message":"2:fetch remote: \"error: object 3b1316b37e444cb7378f09193f556ebee2541cb7: zeroPaddedFilemode: contains zero-padded file modes\\nfatal: fsck error in packed object\\nfatal: fetch-pack: invalid index-pack output\\n\": exit status 128.","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:46:43.370Z","correlation_id":"cf1314c4138bd9afcb1ad48eb907c5b3","message":"starting stage","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:46:43.379Z","correlation_id":"cf1314c4138bd9afcb1ad48eb907c5b3","message":"starting importer","importer":"Importer::RepositoryImporter","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"ERROR","time":"2021-07-08T12:46:51.158Z","correlation_id":"cf1314c4138bd9afcb1ad48eb907c5b3","message":"stage failed","error.message":"2:fetch remote: \"error: object 3b1316b37e444cb7378f09193f556ebee2541cb7: zeroPaddedFilemode: contains zero-padded file modes\\nfatal: fsck error in packed object\\nfatal: fetch-pack: invalid index-pack output\\n\": exit status 128.","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:49:39.248Z","correlation_id":"62ef8bf8f7423b29dd6df7d4b5e63c68","message":"starting stage","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:49:39.259Z","correlation_id":"62ef8bf8f7423b29dd6df7d4b5e63c68","message":"starting importer","importer":"Importer::RepositoryImporter","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"ERROR","time":"2021-07-08T12:49:47.146Z","correlation_id":"62ef8bf8f7423b29dd6df7d4b5e63c68","message":"stage failed","error.message":"2:fetch remote: \"error: object 3b1316b37e444cb7378f09193f556ebee2541cb7: zeroPaddedFilemode: contains zero-padded file modes\\nfatal: fsck error in packed object\\nfatal: fetch-pack: invalid index-pack output\\n\": exit status 128.","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:55:53.664Z","correlation_id":"eb8dcd03758482f0f54dcab2192330a4","message":"starting stage","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"INFO","time":"2021-07-08T12:55:53.680Z","correlation_id":"eb8dcd03758482f0f54dcab2192330a4","message":"starting importer","importer":"Importer::RepositoryImporter","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"ERROR","time":"2021-07-08T12:56:01.458Z","correlation_id":"eb8dcd03758482f0f54dcab2192330a4","message":"stage failed","error.message":"2:fetch remote: \"error: object 3b1316b37e444cb7378f09193f556ebee2541cb7: zeroPaddedFilemode: contains zero-padded file modes\\nfatal: fsck error in packed object\\nfatal: fetch-pack: invalid index-pack output\\n\": exit status 128.","import_source":"github","project_id":644,"import_stage":"Gitlab::GithubImport::Stage::ImportRepositoryWorker"} {"severity":"ERROR","time":"2021-07-08T12:56:01.460Z","correlation_id":null,"event":"github_importer_exhausted","message":"2:fetch remote: \"error: object 3b1316b37e444cb7378f09193f556ebee2541cb7: zeroPaddedFilemode: contains zero-padded file modes\\nfatal: fsck error in packed object\\nfatal: fetch-pack: invalid index-pack output\\n\": exit status 128.","class":"Gitlab::GithubImport::Stage::ImportRepositoryWorker","args":[644],"exception_message":"2:fetch remote: \"error: object 3b1316b37e444cb7378f09193f556ebee2541cb7: zeroPaddedFilemode: contains zero-padded file modes\\nfatal: fsck error in packed object\\nfatal: fetch-pack: invalid index-pack output\\n\": exit status 128.","exception_backtrace":["/srv/gitlab/lib/gitlab/git/wraps_gitaly_errors.rb:15:in `rescue in wrapped_gitaly_errors'","/srv/gitlab/lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'","/srv/gitlab/lib/gitlab/metrics/instrumentation.rb:162:in `wrapped_gitaly_errors'","/srv/gitlab/lib/gitlab/git/repository.rb:820:in `fetch_remote'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/core_ext/module/delegation.rb:310:in `public_send'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/core_ext/module/delegation.rb:310:in `method_missing'","/srv/gitlab/lib/gitlab/metrics/instrumentation.rb:162:in `method_missing'","/srv/gitlab/app/models/repository.rb:941:in `fetch_as_mirror'","/srv/gitlab/lib/gitlab/metrics/instrumentation.rb:162:in `fetch_as_mirror'","/srv/gitlab/lib/gitlab/github_import/importer/repository_importer.rb:53:in `import_repository'","/srv/gitlab/lib/gitlab/github_import/importer/repository_importer.rb:37:in `execute'","/srv/gitlab/app/workers/gitlab/github_import/stage/import_repository_worker.rb:29:in `import'","/srv/gitlab/app/workers/concerns/gitlab/github_import/stage_methods.rb:24:in `try_import'","/srv/gitlab/app/workers/concerns/gitlab/github_import/stage_methods.rb:14:in `perform'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:192:in `execute_job'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'","/srv/gitlab/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:41:in `perform'","/srv/gitlab/lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'","/srv/gitlab/lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `block in call'","/srv/gitlab/lib/gitlab/application_context.rb:74:in `block in use'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.18.0/lib/labkit/context.rb:36:in `with_context'","/srv/gitlab/lib/gitlab/application_context.rb:74:in `use'","/srv/gitlab/lib/gitlab/application_context.rb:27:in `with_context'","/srv/gitlab/lib/gitlab/sidekiq_middleware/worker_context/server.rb:15:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.18.0/lib/labkit/middleware/sidekiq/server.rb:21:in `block in call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.18.0/lib/labkit/middleware/sidekiq/context/server.rb:15:in `block in call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.18.0/lib/labkit/context.rb:36:in `with_context'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.18.0/lib/labkit/middleware/sidekiq/context/server.rb:14:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:133:in `invoke'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.18.0/lib/labkit/middleware/sidekiq/server.rb:20:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","/srv/gitlab/lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","/srv/gitlab/lib/gitlab/with_request_store.rb:10:in `with_request_store'","/srv/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/arguments_logger.rb:8:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/server_metrics.rb:69:in `block in call'","/srv/gitlab/lib/gitlab/sidekiq_middleware/server_metrics.rb:92:in `block in instrument'","/srv/gitlab/lib/gitlab/metrics/background_transaction.rb:30:in `run'","/srv/gitlab/lib/gitlab/sidekiq_middleware/server_metrics.rb:92:in `instrument'","/srv/gitlab/lib/gitlab/sidekiq_middleware/server_metrics.rb:68:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/monitor.rb:8:in `block in call'","/srv/gitlab/lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'","/srv/gitlab/lib/gitlab/sidekiq_middleware/monitor.rb:7:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.10.0/lib/marginalia/sidekiq_instrumentation.rb:10:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-raven-3.1.2/lib/raven/integrations/sidekiq/cleanup_middleware.rb:7:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:133:in `invoke'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:164:in `block in process'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/job_retry.rb:109:in `local'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/rails.rb:43:in `block in call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/execution_wrapper.rb:88:in `wrap'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/reloader.rb:72:in `block in wrap'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/execution_wrapper.rb:88:in `wrap'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/reloader.rb:71:in `wrap'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/rails.rb:42:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:250:in `stats'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/job_logger.rb:8:in `call'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/job_retry.rb:74:in `global'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:125:in `block in dispatch'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/logging.rb:48:in `with_context'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/logging.rb:42:in `with_job_hash_context'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:124:in `dispatch'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:163:in `process'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:83:in `process_one'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:71:in `run'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/util.rb:16:in `watchdog'","/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/util.rb:25:in `block in safe_thread'"]}
Status check via curl
❯ curl -s --header "PRIVATE-TOKEN: ******" "https://gitlab-review-acunskis-t-br5urk.gitlab-review.app/api/v4/projects/644/import" | jq { "id": 644, "description": "Ruby on Rails", "name": "imported-project-dfee691315204bfb", "name_with_namespace": "gitlab-qa-sandbox-group / qa-test-2021-07-08-15-44-13-0cade871bb0fb87c / imported-project-dfee691315204bfb", "path": "imported-project-dfee691315204bfb", "path_with_namespace": "gitlab-qa-sandbox-group/qa-test-2021-07-08-15-44-13-0cade871bb0fb87c/imported-project-dfee691315204bfb", "created_at": "2021-07-08T12:44:20.463Z", "import_status": "started", "correlation_id": "01FA32BGXJKF46C2269KB1JCKF", "failed_relations": [], "import_error": null }
Edited by 🤖 GitLab Bot 🤖