Sentry: GRPC::Internal re git fsck
https://sentry.gitlab.net/gitlab/gitlabcom/issues/4040763/?query=is%3Aunresolved%20feature_category%3Agitaly shows that this is happening rather frequently. I'm copying data here for faster retrieval. Could be corrupted repositories with one example at https://gitlab.com/onap/oom/oom?
How is this related to moving repository maintenance to Gitaly (&7442 (closed), &9768 (comment 1255428132))
GRPC::Internal
13:rescuing dangling objects: git fsck: exit status 16.
grpc_debug_error_string
{"created":"@1674758133.944717161","description":"Error received from peer ipv4:10.221.2.105:9999","file":"src/core/lib/surface/call.cc","file_line":1063,"grpc_message":"rescuing dangling objects: git fsck: exit status 16","grpc_status":13}
server
{os: {build: 5.10.147+, kernel_version: #1 (closed) SMP Sat Oct 22 14:54:40 UTC 2022, name: Linux, version: #1 (closed) SMP Sat Oct 22 14:54:40 UTC 2022}, runtime: {name: ruby, version: ruby 2.7.7p221 (2022-11-24 revision 168ec2b1e5) [x86_64-linux]}}
sidekiq
{args: [29161705, gc, projects_housekeeping:29161705, a4c26bda-bb9a-4b5e-bb3f-02539430abc9], class: Projects::GitGarbageCollectWorker, correlation_id: a6156e632fc5070ae65f8626192813ea, created_at: 1674758133.1310267, enqueued_at: 1674758133.1322248, idempotency_key: ********, jid: fff274af5c285d427ad84234, meta.caller_id: RepositoryUpdateMirrorWorker, meta.client_id: ip/, meta.feature_category: gitaly, meta.project: onap/oom/oom, meta.remote_ip: 140.82.115.96, meta.root_caller_id: POST /api/:version/projects/:id/mirror/pull, meta.root_namespace: onap, queue: default, retry: False, size_limiter: validated, version: 0, worker_data_consistency: always}
GRPC::Internal: 13:rescuing dangling objects: git fsck: exit status 16.
from grpc-1.42.0-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status'
from grpc-1.42.0-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
from grpc-1.42.0-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:376:in `request_response'
from grpc-1.42.0-x86_64 (linux) src/ruby/lib/grpc/generic/client_stub.rb:180:in `block in request_response'
from grpc-1.42.0-x86_64 (linux) src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
from grpc-1.42.0-x86_64 (linux) src/ruby/lib/grpc/generic/client_stub.rb:179:in `request_response'
from grpc-1.42.0-x86_64 (linux) src/ruby/lib/grpc/generic/service.rb:171:in `block (3 levels) in rpc_stub_class'
from lib/gitlab/gitaly_client.rb:185:in `execute'
from lib/gitlab/gitaly_client/call.rb:18:in `block in call'
from lib/gitlab/gitaly_client/call.rb:55:in `recording_request'
from lib/gitlab/gitaly_client/call.rb:17:in `call'
from lib/gitlab/gitaly_client.rb:175:in `call'
from lib/gitlab/gitaly_client/object_pool_service.rb:52:in `block in fetch'
from lib/gitlab/gitaly_client.rb:541:in `with_feature_flag_actors'
from lib/gitlab/gitaly_client/object_pool_service.rb:51:in `fetch'
from lib/gitlab/git/object_pool.rb:44:in `fetch'
from app/services/projects/git_deduplication_service.rb:47:in `fetch_from_source'
from app/services/projects/git_deduplication_service.rb:24:in `block in execute'
from app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'
from app/services/projects/git_deduplication_service.rb:17:in `execute'
from app/workers/projects/git_garbage_collect_worker.rb:23:in `before_gitaly_call'
from app/workers/concerns/git_garbage_collect_methods.rb:34:in `perform'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:202:in `execute_job'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:177:in `block in invoke'
from lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'
from lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform'
from lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'
from lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call'
from lib/gitlab/application_context.rb:115:in `block in use'
from gitlab-labkit (0.29.0) lib/labkit/context.rb:36:in `with_context'
from lib/gitlab/application_context.rb:115:in `use'
from lib/gitlab/application_context.rb:55:in `with_context'
from lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'
from lib/gitlab/database/query_analyzer.rb:37:in `within'
from lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'
from lib/gitlab/with_request_store.rb:17:in `enabling_request_store'
from lib/gitlab/with_request_store.rb:10:in `with_request_store'
from lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'
from lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'
from lib/gitlab/metrics/background_transaction.rb:33:in `run'
from lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'
from lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from gitlab-labkit (0.29.0) lib/labkit/middleware/sidekiq/server.rb:21:in `block in call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:177:in `block in invoke'
from gitlab-labkit (0.29.0) lib/labkit/middleware/sidekiq/context/server.rb:16:in `block in call'
from gitlab-labkit (0.29.0) lib/labkit/context.rb:36:in `with_context'
from gitlab-labkit (0.29.0) lib/labkit/middleware/sidekiq/context/server.rb:15:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:182:in `invoke'
from gitlab-labkit (0.29.0) lib/labkit/middleware/sidekiq/server.rb:20:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'
from lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'
from lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from marginalia (1.11.1) lib/marginalia/sidekiq_instrumentation.rb:9:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from sentry-sidekiq (5.1.1) lib/sentry/sidekiq/sentry_context_middleware.rb:24:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from sentry-raven (3.1.2) lib/raven/integrations/sidekiq/cleanup_middleware.rb:7:in `call'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
from sidekiq (6.5.7) lib/sidekiq/middleware/chain.rb:182:in `invoke'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:169:in `block in process'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
from sidekiq (6.5.7) lib/sidekiq/job_retry.rb:113:in `local'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
from sidekiq (6.5.7) lib/sidekiq/rails.rb:14:in `block in call'
from activesupport (6.1.7.1) lib/active_support/execution_wrapper.rb:91:in `wrap'
from activesupport (6.1.7.1) lib/active_support/reloader.rb:72:in `block in wrap'
from activesupport (6.1.7.1) lib/active_support/execution_wrapper.rb:91:in `wrap'
from activesupport (6.1.7.1) lib/active_support/reloader.rb:71:in `wrap'
from sidekiq (6.5.7) lib/sidekiq/rails.rb:13:in `call'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:263:in `stats'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
from lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
from sidekiq (6.5.7) lib/sidekiq/job_retry.rb:80:in `global'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:124:in `block in dispatch'
from sidekiq (6.5.7) lib/sidekiq/job_logger.rb:39:in `prepare'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:123:in `dispatch'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:168:in `process'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:78:in `process_one'
from sidekiq (6.5.7) lib/sidekiq/processor.rb:68:in `run'
from sidekiq (6.5.7) lib/sidekiq/component.rb:8:in `watchdog'
from sidekiq (6.5.7) lib/sidekiq/component.rb:17:in `block in safe_thread'