Remove a member from a group API in a large namespace against a user that has many memberships can results in an error 500

Summary

Using the Remove a member from a group API in a large namespace against a user that has many memberships can result in an error 500.

The logs show that this is caused by a DB timeout.

This does look like a variation of #467281 (closed)

Steps to reproduce

1. in a large namespace (in this example the namespace contains tens of thousands projects) add a user to many subgroups/projects as member
2. use the Remove a member from a group API against the top level namespace against this user
3. observe the 500 error

Example Project

What is the current bug behavior?

The API will fail with an error 500 and the user's memberships are not removed.

What is the expected correct behavior?

The API should complete and remove the user's memberships.

Relevant logs and/or screenshots

The request fails with an error 500 and the logs for this call show:

• exception.class:Rack::Timeout::RequestTimeoutException
• exception.message:Request ran for longer than 60000ms

The full stack trace is

Click to expand

activerecord (7.1.5.2) lib/active_record/connection_adapters/postgresql_adapter.rb:894:in `exec_params',
activerecord (7.1.5.2) lib/active_record/connection_adapters/postgresql_adapter.rb:894:in `block (2 levels) in exec_no_cache',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract_adapter.rb:1027:in `block in with_raw_connection',
activesupport (7.1.5.2) lib/active_support/concurrency/null_lock.rb:9:in `synchronize',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract_adapter.rb:999:in `with_raw_connection',
activerecord (7.1.5.2) lib/active_record/connection_adapters/postgresql_adapter.rb:893:in `block in exec_no_cache',
activesupport (7.1.5.2) lib/active_support/notifications/instrumenter.rb:58:in `instrument',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract_adapter.rb:1142:in `log',
activerecord (7.1.5.2) lib/active_record/connection_adapters/postgresql_adapter.rb:892:in `exec_no_cache',
activerecord (7.1.5.2) lib/active_record/connection_adapters/postgresql_adapter.rb:872:in `execute_and_clear',
marginalia (1.11.1) lib/marginalia.rb:91:in `execute_and_clear_with_marginalia',
activerecord (7.1.5.2) lib/active_record/connection_adapters/postgresql/database_statements.rb:64:in `internal_exec_query',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/database_statements.rb:630:in `select',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/database_statements.rb:71:in `select_all',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/query_cache.rb:112:in `block in select_all',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/query_cache.rb:152:in `block in cache_sql',
activesupport (7.1.5.2) lib/active_support/concurrency/null_lock.rb:9:in `synchronize',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/query_cache.rb:147:in `cache_sql',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/query_cache.rb:112:in `select_all',
lib/gitlab/database/load_balancing/connection_proxy.rb:107:in `public_send',
lib/gitlab/database/load_balancing/connection_proxy.rb:107:in `block in read_using_load_balancer',
lib/gitlab/database/load_balancing/load_balancer.rb:150:in `block in read_write',
lib/gitlab/database/load_balancing/load_balancer.rb:248:in `retry_with_backoff',
lib/gitlab/database/load_balancing/load_balancer.rb:140:in `read_write',
lib/gitlab/database/load_balancing/connection_proxy.rb:106:in `read_using_load_balancer',
lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all',
activerecord (7.1.5.2) lib/active_record/relation/calculations.rb:313:in `block in pluck',
activerecord (7.1.5.2) lib/active_record/relation.rb:1028:in `skip_query_cache_if_necessary',
activerecord (7.1.5.2) lib/active_record/relation/calculations.rb:309:in `pluck',
lockbox (1.4.1) lib/lockbox/calculations.rb:4:in `pluck',
app/models/user.rb:2447:in `update_two_factor_requirement',
app/models/member.rb:653:in `block in update_two_factor_requirement',
lib/gitlab/database/query_analyzers/prevent_cross_database_modification.rb:29:in `temporary_ignore_tables_in_transaction',
app/models/member.rb:650:in `update_two_factor_requirement',
activesupport (7.1.5.2) lib/active_support/callbacks.rb:403:in `block in make_lambda',
activesupport (7.1.5.2) lib/active_support/callbacks.rb:239:in `block in halting_and_conditional',
activesupport (7.1.5.2) lib/active_support/callbacks.rb:602:in `block in invoke_after',
activesupport (7.1.5.2) lib/active_support/callbacks.rb:602:in `each',
activesupport (7.1.5.2) lib/active_support/callbacks.rb:602:in `invoke_after',
activesupport (7.1.5.2) lib/active_support/callbacks.rb:111:in `run_callbacks',
activesupport (7.1.5.2) lib/active_support/callbacks.rb:952:in `_run_destroy_callbacks',
activerecord (7.1.5.2) lib/active_record/callbacks.rb:423:in `destroy',
activerecord (7.1.5.2) lib/active_record/transactions.rb:305:in `block in destroy',
activerecord (7.1.5.2) lib/active_record/transactions.rb:365:in `block in with_transaction_returning_status',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/transaction.rb:535:in `block in within_new_transaction',
activesupport (7.1.5.2) lib/active_support/concurrency/null_lock.rb:9:in `synchronize',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/transaction.rb:532:in `within_new_transaction',
activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/database_statements.rb:344:in `transaction',
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send',
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer',
lib/gitlab/database/load_balancing/load_balancer.rb:150:in `block in read_write',
lib/gitlab/database/load_balancing/load_balancer.rb:248:in `retry_with_backoff',
lib/gitlab/database/load_balancing/load_balancer.rb:140:in `read_write',
lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer',
lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction',
activerecord (7.1.5.2) lib/active_record/transactions.rb:361:in `with_transaction_returning_status',
activerecord (7.1.5.2) lib/active_record/transactions.rb:305:in `destroy',
app/services/members/destroy_service.rb:88:in `destroy_member',
app/services/members/destroy_service.rb:28:in `execute',
app/services/members/destroy_service.rb:162:in `block in destroy_group_members',
activerecord (7.1.5.2) lib/active_record/relation/delegation.rb:100:in `each',
activerecord (7.1.5.2) lib/active_record/relation/delegation.rb:100:in `each',
app/services/members/destroy_service.rb:159:in `destroy_group_members',
app/services/members/destroy_service.rb:139:in `delete_subgroup_members',
app/services/members/destroy_service.rb:126:in `delete_subresources',
app/services/members/destroy_service.rb:105:in `delete_member_associations',
app/services/members/destroy_service.rb:93:in `destroy_data_related_to_member',
ee/app/services/ee/members/destroy_service.rb:106:in `destroy_data_related_to_member',
app/services/members/destroy_service.rb:84:in `process_destroy_of_group_owner_member',
app/services/members/destroy_service.rb:26:in `execute',
lib/api/members.rb:177:in `block (4 levels) in <class:Members>',
lib/api/helpers.rb:56:in `destroy_conditionally!',
lib/api/members.rb:176:in `block (3 levels) in <class:Members>',
grape (2.0.0) lib/grape/endpoint.rb:58:in `call',
grape (2.0.0) lib/grape/endpoint.rb:58:in `block (2 levels) in generate_api_method',
activesupport (7.1.5.2) lib/active_support/notifications.rb:208:in `instrument',
grape (2.0.0) lib/grape/endpoint.rb:57:in `block in generate_api_method',
grape (2.0.0) lib/grape/endpoint.rb:328:in `execute',
grape (2.0.0) lib/grape/endpoint.rb:260:in `block in run',
activesupport (7.1.5.2) lib/active_support/notifications.rb:208:in `instrument',
grape (2.0.0) lib/grape/endpoint.rb:240:in `run',
grape (2.0.0) lib/grape/endpoint.rb:316:in `block in build_stack',
grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!',
grape (2.0.0) lib/grape/middleware/base.rb:29:in `call',
grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!',
grape (2.0.0) lib/grape/middleware/base.rb:29:in `call',
lib/gitlab/middleware/ip_address.rb:14:in `block in call',
lib/gitlab/ip_address_state.rb:11:in `with',
lib/gitlab/middleware/ip_address.rb:13:in `call',
grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!',
grape (2.0.0) lib/grape/middleware/base.rb:29:in `call',
grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!',
grape (2.0.0) lib/grape/middleware/base.rb:29:in `call',
lib/api/api_guard.rb:272:in `call',
grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!',
grape (2.0.0) lib/grape/middleware/base.rb:29:in `call',
rack-oauth2 (2.2.1) lib/rack/oauth2/server/resource.rb:20:in `_call',
rack-oauth2 (2.2.1) lib/rack/oauth2/server/resource/bearer.rb:8:in `_call',
rack-oauth2 (2.2.1) lib/rack/oauth2/server/abstract/handler.rb:17:in `call',
grape (2.0.0) lib/grape/middleware/error.rb:39:in `block in call!',
grape (2.0.0) lib/grape/middleware/error.rb:38:in `catch',
grape (2.0.0) lib/grape/middleware/error.rb:38:in `call!',
grape (2.0.0) lib/grape/middleware/base.rb:29:in `call',
grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:60:in `block in call!',
grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:58:in `catch',
grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:58:in `call!',
grape (2.0.0) lib/grape/middleware/base.rb:29:in `call',
rack (2.2.20) lib/rack/head.rb:12:in `call',
grape (2.0.0) lib/grape/endpoint.rb:224:in `call!',
grape (2.0.0) lib/grape/endpoint.rb:218:in `call',
grape (2.0.0) lib/grape/router/route.rb:58:in `exec',
grape (2.0.0) lib/grape/router.rb:120:in `process_route',
grape (2.0.0) lib/grape/router.rb:74:in `block in identity',
grape (2.0.0) lib/grape/router.rb:94:in `transaction',
grape (2.0.0) lib/grape/router.rb:72:in `identity',
grape (2.0.0) lib/grape/router.rb:56:in `block in call',
grape (2.0.0) lib/grape/router.rb:136:in `with_optimization',
grape (2.0.0) lib/grape/router.rb:55:in `call',
grape (2.0.0) lib/grape/api/instance.rb:165:in `call',
grape (2.0.0) lib/grape/api/instance.rb:70:in `call!',
grape (2.0.0) lib/grape/api/instance.rb:65:in `call',
actionpack (7.1.5.2) lib/action_dispatch/routing/mapper.rb:31:in `block in <class:Constraints>',
actionpack (7.1.5.2) lib/action_dispatch/routing/mapper.rb:60:in `serve',
actionpack (7.1.5.2) lib/action_dispatch/journey/router.rb:51:in `block in serve',
config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes',
config/initializers/action_dispatch_journey_router.rb:25:in `map!',
config/initializers/action_dispatch_journey_router.rb:25:in `find_routes',
actionpack (7.1.5.2) lib/action_dispatch/journey/router.rb:32:in `serve',
actionpack (7.1.5.2) lib/action_dispatch/routing/route_set.rb:882:in `call',
gitlab-experiment (1.0.0) lib/gitlab/experiment/middleware.rb:19:in `call',
omniauth (2.1.4) lib/omniauth/strategy.rb:478:in `call_app!',
omniauth-saml (2.2.4) lib/omniauth/strategies/saml.rb:83:in `other_phase',
omniauth (2.1.4) lib/omniauth/strategy.rb:195:in `call!',
omniauth (2.1.4) lib/omniauth/strategy.rb:169:in `call',
flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call',
flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call',
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call',
lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call',
lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled',
lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call',
lib/gitlab/middleware/memory_report.rb:13:in `call',
lib/gitlab/middleware/speedscope.rb:13:in `call',
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call',
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call',
lib/gitlab/etag_caching/middleware.rb:21:in `call',
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call',
lib/gitlab/metrics/web_transaction.rb:46:in `run',
lib/gitlab/metrics/rack_middleware.rb:16:in `call',
lib/gitlab/middleware/go.rb:21:in `call',
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call',
lib/gitlab/database/query_analyzer.rb:83:in `within',
lib/gitlab/middleware/query_analyzer.rb:11:in `call',
lib/ci/job_token/middleware.rb:11:in `call',
batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call',
rack-attack (6.7.0) lib/rack/attack.rb:103:in `call',
apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call',
lib/gitlab/middleware/multipart.rb:176:in `call',
rack-attack (6.7.0) lib/rack/attack.rb:127:in `call',
warden (1.2.9) lib/warden/manager.rb:36:in `block in call',
warden (1.2.9) lib/warden/manager.rb:34:in `catch',
warden (1.2.9) lib/warden/manager.rb:34:in `call',
rack-cors (2.0.2) lib/rack/cors.rb:102:in `call',
rack (2.2.20) lib/rack/tempfile_reaper.rb:15:in `call',
rack (2.2.20) lib/rack/etag.rb:27:in `call',
rack (2.2.20) lib/rack/conditional_get.rb:40:in `call',
rack (2.2.20) lib/rack/head.rb:12:in `call',
actionpack (7.1.5.2) lib/action_dispatch/http/permissions_policy.rb:36:in `call',
actionpack (7.1.5.2) lib/action_dispatch/http/content_security_policy.rb:36:in `call',
lib/gitlab/middleware/read_only/controller.rb:40:in `call',
lib/gitlab/middleware/read_only.rb:18:in `call',
lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call',
rack (2.2.20) lib/rack/session/abstract/id.rb:266:in `context',
rack (2.2.20) lib/rack/session/abstract/id.rb:260:in `call',
lib/gitlab/middleware/secure_headers.rb:11:in `call',
actionpack (7.1.5.2) lib/action_dispatch/middleware/cookies.rb:689:in `call',
lib/gitlab/middleware/same_site_cookies.rb:27:in `call',
actionpack (7.1.5.2) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call',
activesupport (7.1.5.2) lib/active_support/callbacks.rb:101:in `run_callbacks',
actionpack (7.1.5.2) lib/action_dispatch/middleware/callbacks.rb:28:in `call',
sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call',
actionpack (7.1.5.2) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call',
lib/gitlab/middleware/path_traversal_check.rb:40:in `call',
lib/gitlab/middleware/handle_malformed_strings.rb:19:in `call',
lib/gitlab/middleware/json_validation.rb:165:in `call',
sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call',
sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking',
sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking',
sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call',
sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope',
sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope',
sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call',
actionpack (7.1.5.2) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call',
lib/gitlab/middleware/basic_health_check.rb:25:in `call',
lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app',
railties (7.1.5.2) lib/rails/rack/logger.rb:24:in `block in call',
activesupport (7.1.5.2) lib/active_support/tagged_logging.rb:139:in `block in tagged',
activesupport (7.1.5.2) lib/active_support/tagged_logging.rb:39:in `tagged',
activesupport (7.1.5.2) lib/active_support/tagged_logging.rb:139:in `tagged',
activesupport (7.1.5.2) lib/active_support/broadcast_logger.rb:241:in `method_missing',
railties (7.1.5.2) lib/rails/rack/logger.rb:24:in `call',
actionpack (7.1.5.2) lib/action_dispatch/middleware/remote_ip.rb:92:in `call',
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call',
lib/gitlab/middleware/request_context.rb:15:in `call',
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call',
request_store (1.7.0) lib/request_store/middleware.rb:19:in `call',
rack (2.2.20) lib/rack/method_override.rb:24:in `call',
rack (2.2.20) lib/rack/runtime.rb:22:in `call',
rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call',
rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout',
rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call',
config/initializers/fix_local_cache_middleware.rb:11:in `call',
lib/gitlab/middleware/compressed_json.rb:44:in `call',
actionpack (7.1.5.2) lib/action_dispatch/middleware/executor.rb:14:in `call',
lib/gitlab/middleware/rack_multipart_tempfile_facto
ry.rb:19:in `call',
lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call',
gitlab-labkit (0.42.0) lib/labkit/middleware/rack.rb:22:in `block in call',
gitlab-labkit (0.42.0) lib/labkit/context.rb:43:in `with_context',
gitlab-labkit (0.42.0) lib/labkit/middleware/rack.rb:21:in `call',
rack (2.2.20) lib/rack/sendfile.rb:127:in `call',
actionpack (7.1.5.2) lib/action_dispatch/middleware/request_id.rb:28:in `call',
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call',
railties (7.1.5.2) lib/rails/engine.rb:536:in `call',
railties (7.1.5.2) lib/rails/railtie.rb:226:in `public_send',
railties (7.1.5.2) lib/rails/railtie.rb:226:in `method_missing',
lib/gitlab/middleware/release_env.rb:12:in `call',
rack (2.2.20) lib/rack/urlmap.rb:74:in `block in call',
rack (2.2.20) lib/rack/urlmap.rb:58:in `each',
rack (2.2.20) lib/rack/urlmap.rb:58:in `call',
puma (6.6.1) lib/puma/configuration.rb:279:in `call',
puma (6.6.1) lib/puma/request.rb:99:in `block in handle_request',
puma (6.6.1) lib/puma/thread_pool.rb:390:in `with_force_shutdown',
puma (6.6.1) lib/puma/request.rb:98:in `handle_request',
puma (6.6.1) lib/puma/server.rb:472:in `process_client',
puma (6.6.1) lib/puma/server.rb:254:in `block in run',
puma (6.6.1) lib/puma/thread_pool.rb:167:in `block in spawn_thread'

Output of checks

Results of GitLab environment info

This was reported by a customer on GitLab Dedicated running 18.5.3

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.