Geo: Error parsing response from node status API
There has been a large number of these errors since 3 days ago: https://sentry.gitlap.com/gitlab/gitlabcom/issues/192214/
TypeError: no implicit conversion of String into Hash
from geo/node_status_fetch_service.rb:14:in `merge!'
from geo/node_status_fetch_service.rb:14:in `call'
from geo/metrics_update_service.rb:58:in `node_status'
from geo/metrics_update_service.rb:28:in `fetch_geo_node_metrics'
from geo/metrics_update_service.rb:18:in `block in fetch_secondary_geo_nodes_metrics'
from active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
from active_record/relation/batches.rb:51:in `each'
from active_record/relation/batches.rb:51:in `block in find_each'
from active_record/relation/batches.rb:124:in `find_in_batches'
from active_record/relation/batches.rb:50:in `find_each'
from geo/metrics_update_service.rb:18:in `fetch_secondary_geo_nodes_metrics'
from geo/metrics_update_service.rb:9:in `execute'
from geo/metrics_update_worker.rb:10:in `block in perform'
from exclusive_lease_guard.rb:22:in `try_obtain_lease'
from geo/metrics_update_worker.rb:10:in `perform'
from sidekiq/processor.rb:188:in `execute_job'
from sidekiq/processor.rb:170:in `block (2 levels) in process'
from sidekiq/middleware/chain.rb:128:in `block in invoke'
from gitlab/sidekiq_status/server_middleware.rb:5:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from gitlab/metrics/sidekiq_middleware.rb:13:in `block in call'
from gitlab/metrics/transaction.rb:53:in `run'
from gitlab/metrics/sidekiq_middleware.rb:13:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from sidekiq/middleware/server/active_record.rb:15:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from raven/integrations/sidekiq.rb:9:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from sidekiq/middleware/chain.rb:133:in `invoke'
from sidekiq/processor.rb:169:in `block in process'
from sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
from sidekiq/job_retry.rb:97:in `local'
from sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
from sidekiq.rb:36:in `block in <module:Sidekiq>'
from sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
from sidekiq/processor.rb:204:in `stats'
from sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
from gitlab/sidekiq_logging/structured_logger.rb:13:in `call'
from sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
from sidekiq/job_retry.rb:72:in `global'
from sidekiq/processor.rb:129:in `block in dispatch'
from sidekiq/logging.rb:44:in `with_context'
from sidekiq/logging.rb:38:in `with_job_hash_context'
from sidekiq/processor.rb:128:in `dispatch'
from sidekiq/processor.rb:168:in `process'
from sidekiq/processor.rb:85:in `process_one'
from sidekiq/processor.rb:73:in `run'
from sidekiq/util.rb:16:in `watchdog'
from sidekiq/util.rb:25:in `block in safe_thread'
This seems like a 200 response was sent, but with a string message?