GRPC::DeadlineExceeded exception when removing blobs to reduce repository size
Summary
When trying to remove blobs to reduce repository size, request is timing out after 54 seconds.
Steps to reproduce
Follow steps in Remove blobs.
What is the current bug behavior?
On the client side, an error is received that reads "Something went wrong while removing blobs."
In the logs, there is an exception with backtrace:
[grpc-1.63.0-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status', grpc-1.63.0-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:189:in `attach_status_results_and_complete_call', grpc-1.63.0-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:424:in `client_streamer', grpc-1.63.0-x86_64 (linux) src/ruby/lib/grpc/generic/client_stub.rb:257:in `block in client_streamer', grpc-1.63.0-x86_64 (linux) src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!', grpc-1.63.0-x86_64 (linux) src/ruby/lib/grpc/generic/client_stub.rb:256:in `client_streamer', grpc-1.63.0-x86_64 (linux) src/ruby/lib/grpc/generic/service.rb:176:in `block (3 levels) in rpc_stub_class', lib/gitlab/gitaly_client.rb:291:in `execute', lib/gitlab/gitaly_client/call.rb:18:in `block in call', lib/gitlab/gitaly_client/call.rb:60:in `recording_request', lib/gitlab/gitaly_client/call.rb:17:in `call', lib/gitlab/gitaly_client.rb:280:in `call', lib/gitlab/gitaly_client/with_feature_flag_actors.rb:31:in `block in gitaly_client_call', lib/gitlab/gitaly_client.rb:630:in `with_feature_flag_actors', lib/gitlab/gitaly_client/with_feature_flag_actors.rb:25:in `gitaly_client_call', lib/gitlab/gitaly_client/cleanup_service.rb:45:in `rewrite_history', app/graphql/mutations/projects/blobs_remove.rb:37:in `resolve', graphql (2.3.5) lib/graphql/schema/resolver.rb:119:in `public_send', graphql (2.3.5) lib/graphql/schema/resolver.rb:119:in `call_resolve', graphql (2.3.5) lib/graphql/schema/mutation.rb:69:in `call_resolve', graphql (2.3.5) lib/graphql/schema/resolver.rb:106:in `block (3 levels) in resolve_with_support', graphql (2.3.5) lib/graphql/schema.rb:1365:in `after_lazy', graphql (2.3.5) lib/graphql/query.rb:374:in `after_lazy', graphql (2.3.5) lib/graphql/schema/resolver.rb:94:in `block (2 levels) in resolve_with_support', graphql (2.3.5) lib/graphql/schema.rb:1365:in `after_lazy', graphql (2.3.5) lib/graphql/query.rb:374:in `after_lazy', graphql (2.3.5) lib/graphql/schema/resolver.rb:85:in `block in resolve_with_support', graphql (2.3.5) lib/graphql/schema.rb:1365:in `after_lazy', graphql (2.3.5) lib/graphql/query.rb:374:in `after_lazy', graphql (2.3.5) lib/graphql/schema/resolver.rb:73:in `resolve_with_support', graphql (2.3.5) lib/graphql/schema/has_single_input_argument.rb:36:in `resolve_with_support', graphql (2.3.5) lib/graphql/schema/relay_classic_mutation.rb:45:in `resolve_with_support', graphql (2.3.5) lib/graphql/schema/field.rb:683:in `public_send', graphql (2.3.5) lib/graphql/schema/field.rb:683:in `block (2 levels) in resolve', graphql (2.3.5) lib/graphql/schema/field.rb:828:in `block in with_extensions', graphql (2.3.5) lib/graphql/schema/field.rb:864:in `block (2 levels) in run_extensions_before_resolve', graphql (2.3.5) lib/graphql/schema/field.rb:864:in `block (2 levels) in run_extensions_before_resolve', graphql (2.3.5) lib/graphql/schema/field.rb:867:in `run_extensions_before_resolve', graphql (2.3.5) lib/graphql/schema/field.rb:864:in `block in run_extensions_before_resolve', graphql (2.3.5) lib/graphql/schema/field_extension.rb:134:in `resolve', graphql (2.3.5) lib/graphql/schema/field.rb:851:in `run_extensions_before_resolve', graphql (2.3.5) lib/graphql/schema/field.rb:864:in `block in run_extensions_before_resolve', lib/gitlab/graphql/present/field_extension.rb:18:in `resolve', graphql (2.3.5) lib/graphql/schema/field.rb:851:in `run_extensions_before_resolve', graphql (2.3.5) lib/graphql/schema/field.rb:823:in `with_extensions', graphql (2.3.5) lib/graphql/schema/field.rb:654:in `block in resolve', graphql (2.3.5) lib/graphql/schema.rb:1365:in `after_lazy', graphql (2.3.5) lib/graphql/query.rb:374:in `after_lazy', graphql (2.3.5) lib/graphql/schema/field.rb:652:in `resolve', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:366:in `block (2 levels) in evaluate_selection_with_resolved_keyword_args', graphql (2.3.5) lib/graphql/tracing/trace.rb:52:in `execute_field', batch-loader (2.0.5) lib/batch_loader/graphql.rb:7:in `execute_field', graphql (2.3.5) lib/graphql/schema/timeout.rb:84:in `execute_field', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:365:in `block in evaluate_selection_with_resolved_keyword_args', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:687:in `call_method_on_directives', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:354:in `evaluate_selection_with_resolved_keyword_args', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:332:in `block in evaluate_selection_with_args', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:806:in `after_lazy', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:279:in `evaluate_selection_with_args', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:273:in `block in evaluate_selection', graphql (2.3.5) lib/graphql/execution/interpreter/arguments_cache.rb:46:in `dataload_for', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:271:in `evaluate_selection', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:212:in `block (3 levels) in evaluate_selections', graphql (2.3.5) lib/graphql/dataloader/null_dataloader.rb:19:in `append_job', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:211:in `block (2 levels) in evaluate_selections', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:210:in `each', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:210:in `block in evaluate_selections', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:687:in `call_method_on_directives', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:207:in `evaluate_selections', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:97:in `block (3 levels) in run_eager', graphql (2.3.5) lib/graphql/dataloader/null_dataloader.rb:19:in `append_job', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:96:in `block (2 levels) in run_eager', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:117:in `each_gathered_selections', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:87:in `block in run_eager', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:687:in `call_method_on_directives', graphql (2.3.5) lib/graphql/execution/interpreter/runtime.rb:86:in `run_eager', graphql (2.3.5) lib/graphql/execution/interpreter.rb:70:in `block (4 levels) in run_all', graphql (2.3.5) lib/graphql/tracing/trace.rb:44:in `execute_query', graphql (2.3.5) lib/graphql/execution/interpreter.rb:69:in `block (3 levels) in run_all', graphql (2.3.5) lib/graphql/dataloader/null_dataloader.rb:19:in `append_job', graphql (2.3.5) lib/graphql/execution/interpreter.rb:57:in `block (2 levels) in run_all', graphql (2.3.5) lib/graphql/execution/interpreter.rb:53:in `each', graphql (2.3.5) lib/graphql/execution/interpreter.rb:53:in `each_with_index', graphql (2.3.5) lib/graphql/execution/interpreter.rb:53:in `block in run_all', graphql (2.3.5) lib/graphql/tracing/trace.rb:40:in `execute_multiplex', lib/gitlab/graphql/tracers/instrumentation_tracer.rb:17:in `execute_multiplex', graphql (2.3.5) lib/graphql/schema/timeout.rb:68:in `execute_multiplex', graphql (2.3.5) lib/graphql/execution/interpreter.rb:37:in `run_all', graphql (2.3.5) lib/graphql/schema.rb:1327:in `multiplex', app/graphql/gitlab_schema.rb:44:in `multiplex', graphql (2.3.5) lib/graphql/schema.rb:1303:in `execute', app/controllers/graphql_controller.rb:246:in `execute_query', app/controllers/graphql_controller.rb:73:in `execute', actionpack (7.0.8.4) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action', actionpack (7.0.8.4) lib/abstract_controller/base.rb:215:in `process_action', actionpack (7.0.8.4) lib/action_controller/metal/rendering.rb:165:in `process_action', actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:234:in `block in process_action', activesupport (7.0.8.4) lib/active_support/callbacks.rb:118:in `block in run_callbacks', lib/gitlab/ip_address_state.rb:11:in `with', ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address', activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks', app/controllers/application_controller.rb:474:in `set_current_admin', activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks', lib/gitlab/session.rb:11:in `with_session', app/controllers/application_controller.rb:465:in `set_session_storage', activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks', lib/gitlab/i18n.rb:114:in `with_locale', lib/gitlab/i18n.rb:120:in `with_user_locale', app/controllers/application_controller.rb:456:in `set_locale', activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks', marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment', activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks', app/controllers/application_controller.rb:449:in `set_current_context', activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks', sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:28:in `block in sentry_around_action', sentry-ruby (5.17.3) lib/sentry/hub.rb:102:in `with_child_span', sentry-ruby (5.17.3) lib/sentry-ruby.rb:490:in `with_child_span', sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:14:in `sentry_around_action', activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks', activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks', actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action', actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action', actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action', activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument', activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument', activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument', actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action', actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action', activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action', actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process', actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process', actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch', actionpack (7.0.8.4) lib/action_controller/metal.rb:251:in `dispatch', actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch', actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve', actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve', actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each', actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve', actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call', gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call', omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!', omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call', omniauth (2.1.0) lib/omniauth/strategy.rb:470:in `call_app!', ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase', omniauth (2.1.0) lib/omniauth/strategy.rb:195:in `call!', omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call', omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!', omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call', omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!', omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call', omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!', omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call', flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call', flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call', lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call', lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call', lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled', lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call', lib/gitlab/middleware/memory_report.rb:13:in `call', lib/gitlab/middleware/speedscope.rb:13:in `call', lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call', lib/gitlab/middleware/rails_queue_duration.rb:33:in `call', lib/gitlab/etag_caching/middleware.rb:21:in `call', lib/gitlab/metrics/rack_middleware.rb:16:in `block in call', lib/gitlab/metrics/web_transaction.rb:46:in `run', lib/gitlab/metrics/rack_middleware.rb:16:in `call', lib/gitlab/middleware/go.rb:24:in `call', lib/gitlab/middleware/query_analyzer.rb:11:in `block in call', lib/gitlab/database/query_analyzer.rb:40:in `within', lib/gitlab/middleware/query_analyzer.rb:11:in `call', batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call', rack-attack (6.7.0) lib/rack/attack.rb:103:in `call', apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call', lib/gitlab/middleware/multipart.rb:173:in `call', rack-attack (6.7.0) lib/rack/attack.rb:127:in `call', warden (1.2.9) lib/warden/manager.rb:36:in `block in call', warden (1.2.9) lib/warden/manager.rb:34:in `catch', warden (1.2.9) lib/warden/manager.rb:34:in `call', rack-cors (2.0.2) lib/rack/cors.rb:102:in `call', rack (2.2.9) lib/rack/tempfile_reaper.rb:15:in `call', rack (2.2.9) lib/rack/etag.rb:27:in `call', rack (2.2.9) lib/rack/conditional_get.rb:40:in `call', rack (2.2.9) lib/rack/head.rb:12:in `call', actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call', actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call', lib/gitlab/middleware/read_only/controller.rb:50:in `call', lib/gitlab/middleware/read_only.rb:18:in `call', lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call', rack (2.2.9) lib/rack/session/abstract/id.rb:266:in `context', rack (2.2.9) lib/rack/session/abstract/id.rb:260:in `call', actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call', lib/gitlab/middleware/same_site_cookies.rb:27:in `call', actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call', activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks', actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call', sentry-rails (5.17.3) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call', actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call', lib/gitlab/middleware/path_traversal_check.rb:34:in `call', lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call', sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:29:in `block (2 levels) in call', sentry-ruby (5.17.3) lib/sentry/hub.rb:251:in `with_session_tracking', sentry-ruby (5.17.3) lib/sentry-ruby.rb:403:in `with_session_tracking', sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:20:in `block in call', sentry-ruby (5.17.3) lib/sentry/hub.rb:59:in `with_scope', sentry-ruby (5.17.3) lib/sentry-ruby.rb:383:in `with_scope', sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:19:in `call', actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call', lib/gitlab/middleware/basic_health_check.rb:25:in `call', lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app', railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call', activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged', activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged', activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged', railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call', actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call', lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call', lib/gitlab/middleware/request_context.rb:15:in `call', lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call', request_store (1.5.1) lib/request_store/middleware.rb:19:in `call', rack (2.2.9) lib/rack/method_override.rb:24:in `call', rack (2.2.9) lib/rack/runtime.rb:22:in `call', rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call', rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout', rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call', config/initializers/fix_local_cache_middleware.rb:11:in `call', lib/gitlab/middleware/compressed_json.rb:44:in `call', actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call', lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call', rack (2.2.9) lib/rack/sendfile.rb:110:in `call', lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call', lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call', gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:22:in `block in call', gitlab-labkit (0.36.1) lib/labkit/context.rb:35:in `with_context', gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:21:in `call', actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call', actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call', railties (7.0.8.4) lib/rails/engine.rb:530:in `call', railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send', railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing', lib/gitlab/middleware/release_env.rb:13:in `call', rack (2.2.9) lib/rack/urlmap.rb:74:in `block in call', rack (2.2.9) lib/rack/urlmap.rb:58:in `each', rack (2.2.9) lib/rack/urlmap.rb:58:in `call', puma (6.4.0) lib/puma/configuration.rb:272:in `call', puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request', puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown', puma (6.4.0) lib/puma/request.rb:99:in `handle_request', puma (6.4.0) lib/puma/server.rb:443:in `process_client', puma (6.4.0) lib/puma/server.rb:241:in `block in run', puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread']What is the expected correct behavior?
Removing blob should complete successfully.
Relevant logs and/or screenshots
Kibana logs - internal only.