On GitLab.com, the `CreateNoteDiffFileWorker` sidekiq job has a ~50% failure rate
On GitLab.com, the CreateNoteDiffFileWorker
sidekiq job has about a 50% failure rate.
The error appears to be ActiveRecord::RecordNotFound: Couldn't find DiffNote with 'id'=..
Stacktrace:
app/workers/create_note_diff_file_worker.rb:13:in `perform',
lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:24:in `call',
lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform',
lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:57:in `perform',
lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call',
lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context',
lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `block in call',
lib/gitlab/application_context.rb:74:in `block in use',
lib/gitlab/application_context.rb:74:in `use',
lib/gitlab/application_context.rb:27:in `with_context',
lib/gitlab/sidekiq_middleware/worker_context/server.rb:15:in `call',
lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call',
lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call',
lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call',
lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call',
lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call',
lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call',
lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call',
lib/gitlab/with_request_store.rb:17:in `enabling_request_store',
lib/gitlab/with_request_store.rb:10:in `with_request_store',
lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call',
lib/gitlab/sidekiq_middleware/server_metrics.rb:66:in `block in call',
lib/gitlab/sidekiq_middleware/server_metrics.rb:89:in `block in instrument',
lib/gitlab/metrics/background_transaction.rb:30:in `run',
lib/gitlab/sidekiq_middleware/server_metrics.rb:89:in `instrument',
lib/gitlab/sidekiq_middleware/server_metrics.rb:65:in `call',
lib/gitlab/sidekiq_middleware/monitor.rb:8:in `block in call',
lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job',
lib/gitlab/sidekiq_middleware/monitor.rb:7:in `call',
lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call',
lib/gitlab/sidekiq_logging/structured_logger.rb:19:in `call'
Proposal
Investigate why these jobs are failing, possibly ensure that the entity exists before issuing the sidekiq job, avoiding unnecessary sidekiq load, error noise, etc.
Verification
Check https://log.gprd.gitlab.net/goto/6fe8cab9dc79c0f45ba663fec3c0b18a for high failure rates.
Edited by Andrew Newdigate