Sidekiq BitBucketServerImport job failures with undefined method `credentials'
Summary
On a Dedicated instance running 16.2.7, we are seeing some sidekiq errors on jobs of class Gitlab::BitbucketServerImport::ImportPullRequestNotesWorker
.
The error class is NoMethodError
with the message: undefined method
credentials' for nil:NilClass`.
Backtrace
Seemingly someone is attempting to do a BitBucket Import without credentials.
Could this user error be caught before the sidekiq job kicks off to avoid triggering an apdex SLO violation?
Steps to reproduce
No exact details are available to me as this is reproduced on the client side, but seemingly creating Bit Bucket Import jobs without credentials.
What is the current bug behavior?
When credentials are not passed to the bit bucket importer, the initializer errors and the job fails with a NoMethodError
.
What is the expected correct behavior?
Catch the exception as a 4xx
Relevant logs and/or screenshots
From our logs we have this backtrace:
lib/gitlab/bitbucket_server_import/importers/pull_request_notes_importer.rb:12:in `initialize',
app/workers/concerns/gitlab/bitbucket_server_import/object_importer.rb:53:in `new',
app/workers/concerns/gitlab/bitbucket_server_import/object_importer.rb:53:in `import',
app/workers/concerns/gitlab/bitbucket_server_import/object_importer.rb:41:in `perform',
lib/gitlab/sidekiq_middleware/skip_jobs.rb:49:in `call',
lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in `call',
lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform',
lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform',
lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call',
lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context',
lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call',
lib/gitlab/application_context.rb:124:in `block in use',
lib/gitlab/application_context.rb:124:in `use',
lib/gitlab/application_context.rb:62:in `with_context',
lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call',
lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call',
lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call',
lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call',
lib/gitlab/database/query_analyzer.rb:37:in `within',
lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call',
lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call',
lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call',
lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call',
lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call',
lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call',
lib/gitlab/with_request_store.rb:17:in `enabling_request_store',
lib/gitlab/with_request_store.rb:10:in `with_request_store',
lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call',
lib/gitlab/sidekiq_middleware/server_metrics.rb:84:in `block in call',
lib/gitlab/sidekiq_middleware/server_metrics.rb:111:in `block in instrument',
lib/gitlab/metrics/background_transaction.rb:33:in `run',
lib/gitlab/sidekiq_middleware/server_metrics.rb:111:in `instrument',
lib/gitlab/sidekiq_middleware/server_metrics.rb:83:in `call', lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call',
lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job',
lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call',
lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call',
lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'
Possible fixes
Add a check to ObjectImporter
to return early if the import state is failed. This should fix the case when the import failed but an ObjectImporter
was already enqueued.