Unusual high number of Deadline Exceeded messages from Gitaly
On GPRD sidekiq-asap-01-gprd, I am seeing a flood of these DeadlineExceeded messages from Gitaly (https://log.gitlab.net/goto/b6fef0453c5b6b83b4d2ac4a32dd2537):
{"severity":"INFO","time":"2018-06-14T04:42:39.352Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[1469160],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"996bb4e6b89bdb09373690b6","created_at":"2018-06-14T04:37:21.956Z","enqueued_at":"2018-06-14T04:40:20.872Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951170.5116563,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-996bb4e6b89bdb09373690b6: start","job_status":"start"}
{"severity":"INFO","time":"2018-06-14T04:42:39.371Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[1486668],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"736ce2131e4ba96837b9d0ed","created_at":"2018-06-14T04:37:21.987Z","enqueued_at":"2018-06-14T04:40:20.873Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951173.5433288,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-736ce2131e4ba96837b9d0ed: start","job_status":"start"}
{"severity":"INFO","time":"2018-06-14T04:42:39.396Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[591785],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"f9df3c52d6b22736b8b7fb32","created_at":"2018-06-14T04:37:21.962Z","enqueued_at":"2018-06-14T04:40:20.874Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951170.766547,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-f9df3c52d6b22736b8b7fb32: start","job_status":"start"}
{"severity":"INFO","time":"2018-06-14T04:42:39.411Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[766380],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"296422adab9cbb0c04f6e829","created_at":"2018-06-14T04:37:22.003Z","enqueued_at":"2018-06-14T04:40:20.875Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951178.7837062,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-296422adab9cbb0c04f6e829: start","job_status":"start"}
{"severity":"INFO","time":"2018-06-14T04:42:39.446Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[1490487],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"11cb23e39fa43806795ca2bd","created_at":"2018-06-14T04:37:22.041Z","enqueued_at":"2018-06-14T04:40:20.875Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951181.0212748,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-11cb23e39fa43806795ca2bd: start","job_status":"start"}
{"severity":"INFO","time":"2018-06-14T04:42:39.485Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[1485933],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"817aedc01814c9ba5fbd2094","created_at":"2018-06-14T04:37:22.081Z","enqueued_at":"2018-06-14T04:40:20.876Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951186.6290526,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-817aedc01814c9ba5fbd2094: start","job_status":"start"}
{"severity":"INFO","time":"2018-06-14T04:42:39.488Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[1400955],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"356431a67dfb32ed9705059c","created_at":"2018-06-14T04:37:22.008Z","enqueued_at":"2018-06-14T04:40:20.878Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951179.1757364,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-356431a67dfb32ed9705059c: start","job_status":"start"}
{"severity":"INFO","time":"2018-06-14T04:42:39.503Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[1486605],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"234d74a037c305e365499e14","created_at":"2018-06-14T04:37:21.996Z","enqueued_at":"2018-06-14T04:40:20.879Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951175.3669744,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-234d74a037c305e365499e14: start","job_status":"start"}
{"severity":"INFO","time":"2018-06-14T04:42:39.528Z","class":"Geo::RepositoryVerification::Secondary::SingleWorker","args":[1495596],"retry":3,"queue":"geo:geo_repository_verification_secondary_single","queue_namespace":"geo","jid":"f7f0606b1237a38040b00b5d","created_at":"2018-06-14T04:37:21.978Z","enqueued_at":"2018-06-14T04:40:20.879Z","error_message":"4:Deadline Exceeded","error_class":"GRPC::DeadlineExceeded","failed_at":1528951171.4705036,"retry_count":0,"pid":32015,"message":"Geo::RepositoryVerification::Secondary::SingleWorker JID-f7f0606b1237a38040b00b5d: start","job_status":"start"}I tried to see if there was any correlation with NFS servers, but I couldn't find any. Some of these projects don't exist. The Gitaly logs for the file servers I checked didn't show anything interesting.
I'm running gitlab-ctl configure and sudo gitlab-ctl restart gitaly to see if this helps.
/cc: @dbalexandre