Elastic::ProjectTransferWorker should handle missing namespace
Summary
A increased rate of errors have been reported from the Elastic:::ProjectTransferWorker recently
Kibana logs (internal only) show that there are repeated errors in looking up the old namespace
Couldn't find Namespace with 'id'=XXXXXX
Click to expand backtrace
ee/app/workers/elastic/project_transfer_worker.rb:14:in `perform', lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call', lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:58:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call', lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call', lib/gitlab/application_context.rb:113:in `block in use', lib/gitlab/application_context.rb:113:in `use', lib/gitlab/application_context.rb:54:in `with_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call', lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call', lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call', lib/gitlab/database/query_analyzer.rb:37:in `within', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call', lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call', lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call', lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call', lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call', lib/gitlab/with_request_store.rb:17:in `enabling_request_store', lib/gitlab/with_request_store.rb:10:in `with_request_store', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call', lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument', lib/gitlab/metrics/background_transaction.rb:33:in `run', lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument', lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call', lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call', lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job', lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call', lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call', lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'
Steps to reproduce
Not confirmed, but I think this could reproduce the error
- transfer a project from namespace A to namespace B
- quickly delete namespace A
- view sidekiq logs
Possible fixes
project_transfer_worker.rb: Gracefully handle if the old namespace is gone. I suspect the namespace was deleted after project transfer was reported successfully in the UI to the user, but before any background workers have ran or completed.
If the old_namespace is not found, the code should:
- invalidate the elasticsearch indexing cache if elasticsearch_limit_indexing is enabled
- perform the remaining work of indexing or deleting from the index once the cache is cleared