Unable to push NPM package to only 1 repo/registry

Summary

This randomly started a few days ago on a fully updated self-hosted Gitlab instance. There is 1 NPM registry that we cannot publish any packages into.

This happens on pipelines, and 4 Local development environments. I have pushed the package to another repo and it worked fine, so the issue appears to be limited to only this one package registry/repo

Steps to reproduce

• run yarn build in order to build the project
• run npm publish to publish the tar.gz to Gitlab
• After a long time, yarn errors with a 500 error.

What is the current bug behavior?

On one NPM package registry we are unable to publish any packages. It has worked twice randomly over about 50 tries in the last 2 days as we troubleshot.

What is the expected correct behavior?

You should be able to publish this package to the package registry

Relevant logs and/or screenshots

Error in Yarn:

npm notice 
npm notice Publishing to https://src.casetivity.com/api/v4/projects/76/packages/npm/
npm ERR! code E500
npm ERR! 500 Internal Server Error - PUT https://src.casetivity.com/api/v4/projects/76/packages/npm/@ssg%2fcasetivity-front-end

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/hjamison/.npm/_logs/2024-01-12T04_27_55_981Z-debug-0.log

Error from Gitlab production log:

ArgumentError (Too many Dependencies selected):
  /opt/gitlab/embedded/service/gitlab-rails/app/models/packages/dependency.rb:28:in `block in ids_for_package_names_and_version_patterns'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/packages/dependency.rb:20:in `each_slice'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/packages/dependency.rb:20:in `ids_for_package_names_and_version_patterns'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/packages/dependency.rb:35:in `for_package_names_and_version_patterns'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/create_dependency_service.rb:38:in `find_existing_ids_and_names'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/create_dependency_service.rb:24:in `create_dependency'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/create_dependency_service.rb:14:in `block in execute'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/create_dependency_service.rb:13:in `each_key'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/create_dependency_service.rb:13:in `execute'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/npm/create_package_service.rb:33:in `create_npm_package!'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/npm/create_package_service.rb:19:in `block (2 levels) in execute'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8/lib/active_record/transactions.rb:209:in `transaction'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:359:in `block in transaction'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:358:in `transaction'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:83:in `transaction'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/npm/create_package_service.rb:19:in `block in execute'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/packages/npm/create_package_service.rb:18:in `execute'
  /opt/gitlab/embedded/service/gitlab-rails/lib/api/npm_project_packages.rb:78:in `block (2 levels) in <class:NpmProjectPackages>'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:58:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:58:in `block (2 levels) in generate_api_method'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:208:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:57:in `block in generate_api_method'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:328:in `execute'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:261:in `block in run'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:208:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:241:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:316:in `block in build_stack'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:36:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:29:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:36:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:29:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:36:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:29:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/api/api_guard.rb:219:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:36:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:29:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-oauth2-1.21.3/lib/rack/oauth2/server/resource.rb:20:in `_call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-oauth2-1.21.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-oauth2-1.21.3/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/error.rb:39:in `block in call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/error.rb:38:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/error.rb:38:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:29:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape_logging-1.8.4/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape_logging-1.8.4/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape_logging-1.8.4/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/middleware/base.rb:29:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:225:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/endpoint.rb:219:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/router/route.rb:58:in `exec'
opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/router.rb:120:in `process_route'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/router.rb:74:in `block in identity'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/router.rb:94:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/router.rb:72:in `identity'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/router.rb:56:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/router.rb:136:in `with_optimization'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/router.rb:55:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/api/instance.rb:165:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/api/instance.rb:70:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/grape-1.7.1/lib/grape/api/instance.rb:65:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/routing/mapper.rb:48:in `serve'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:50:in `block in serve'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:32:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:32:in `serve'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/routing/route_set.rb:852:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/gitlab-experiment-0.9.1/lib/gitlab/experiment/middleware.rb:19:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omniauth-2.1.0/lib/omniauth/strategy.rb:470:in `call_app!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omniauth-saml-2.1.0/lib/omniauth/strategies/saml.rb:86:in `other_phase'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omniauth-2.1.0/lib/omniauth/strategy.rb:195:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omniauth-2.1.0/lib/omniauth/strategy.rb:169:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/flipper-0.26.2/lib/flipper/middleware/memoizer.rb:72:in `memoized_call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/flipper-0.26.2/lib/flipper/middleware/memoizer.rb:37:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/memory_report.rb:13:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/speedscope.rb:13:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:21:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/query_analyzer.rb:11:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/batch-loader-2.0.1/lib/batch_loader/middleware.rb:11:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-attack-6.7.0/lib/rack/attack.rb:103:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/apollo_upload_server-2.1.5/lib/apollo_upload_server/middleware.rb:19:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:173:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-attack-6.7.0/lib/rack/attack.rb:127:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/warden-1.2.9/lib/warden/manager.rb:36:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-cors-2.0.1/lib/rack/cors.rb:102:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:15:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/etag.rb:27:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/conditional_get.rb:40:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/http/permissions_policy.rb:38:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/http/content_security_policy.rb:36:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:50:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266:in `context'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/cookies.rb:704:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:99:in `run_callbacks'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-rails-5.10.0/lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rack/capture_exceptions.rb:28:in `block (2 levels) in call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/hub.rb:227:in `with_session_tracking'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry-ruby.rb:385:in `with_session_tracking'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rack/capture_exceptions.rb:19:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/hub.rb:59:in `with_scope'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry-ruby.rb:365:in `with_scope'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rack/capture_exceptions.rb:18:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/lograge-0.11.2/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/rack/logger.rb:25:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:99:in `block in tagged'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:37:in `tagged'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:99:in `tagged'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/rack/logger.rb:25:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/request_context.rb:15:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/method_override.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/runtime.rb:22:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-timeout-0.6.3/lib/rack/timeout/core.rb:148:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/timeout.rb:19:in `timeout'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-timeout-0.6.3/lib/rack/timeout/core.rb:147:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:11:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/compressed_json.rb:44:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/executor.rb:14:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/sendfile.rb:110:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/gitlab-labkit-0.34.0/lib/labkit/middleware/rack.rb:22:in `block in call'
opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/gitlab-labkit-0.34.0/lib/labkit/context.rb:35:in `with_context'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/gitlab-labkit-0.34.0/lib/labkit/middleware/rack.rb:21:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/request_id.rb:26:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-raven-3.1.2/lib/raven/integrations/rack.rb:51:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/engine.rb:530:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/railtie.rb:226:in `public_send'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/railtie.rb:226:in `method_missing'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:13:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/configuration.rb:272:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/request.rb:100:in `block in handle_request'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/thread_pool.rb:378:in `with_force_shutdown'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/request.rb:99:in `handle_request'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/server.rb:443:in `process_client'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/server.rb:241:in `block in run'
  /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/thread_pool.rb:155:in `block in spawn_thread'

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   3.1.4p223
Gem Version:    3.4.22
Bundler Version:2.4.22
Rake Version:   13.0.6
Redis Version:  7.0.14
Sidekiq Version:6.5.12
Go Version:     unknown

GitLab information
Version:        16.7.0
Revision:       0fb7483d0df
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     14.9
URL:            https://src.casetivity.com
HTTP Clone URL: https://src.casetivity.com/some-group/some-project.git
SSH Clone URL:  git@src.casetivity.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml

GitLab Shell
Version:        14.32.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      16.7.0
- default Git Version:  2.42.0

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.32.0 ? ... OK (14.32.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
36/4 ... yes
36/6 ... yes
46/7 ... yes
44/8 ... yes
46/9 ... yes
46/10 ... yes
46/11 ... yes
34/12 ... yes
34/13 ... yes
35/16 ... yes
34/17 ... yes
35/19 ... yes
52/20 ... yes
35/22 ... yes
35/26 ... yes
54/31 ... yes
35/32 ... yes
54/36 ... yes
35/37 ... yes
52/38 ... yes
52/40 ... yes
35/41 ... yes
35/42 ... yes
54/43 ... yes
35/44 ... yes
54/45 ... yes
46/47 ... yes
58/48 ... yes
58/49 ... yes
38/50 ... yes
52/51 ... yes
38/52 ... yes
35/53 ... yes
63/56 ... yes
63/57 ... yes
46/59 ... yes
58/60 ... yes
35/61 ... yes
63/62 ... yes
52/64 ... yes
35/65 ... yes
35/72 ... yes
46/73 ... yes
46/74 ... yes
35/75 ... yes
35/76 ... yes
35/77 ... yes
35/78 ... yes
35/79 ... yes
35/80 ... yes
58/81 ... yes
46/83 ... yes
79/85 ... yes
79/86 ... yes
35/87 ... yes
52/88 ... yes
47/89 ... yes
35/90 ... yes
35/91 ... yes
82/92 ... yes
82/93 ... yes
82/94 ... yes
82/95 ... yes
35/96 ... yes
52/97 ... yes
52/98 ... yes
35/99 ... yes
35/100 ... yes
35/101 ... yes
35/102 ... yes
35/103 ... yes
35/104 ... yes
35/105 ... yes
35/106 ... yes
35/107 ... yes
35/108 ... yes
122/109 ... yes
122/111 ... yes
124/113 ... yes
124/114 ... yes
124/115 ... yes
124/116 ... yes
124/118 ... yes
124/119 ... yes
122/121 ... yes
122/122 ... yes
122/124 ... yes
122/125 ... yes
122/126 ... yes
122/127 ... yes
122/128 ... yes
122/129 ... yes
122/130 ... yes
122/131 ... yes
122/132 ... yes
122/133 ... yes
122/134 ... yes
122/136 ... yes
122/137 ... yes
121/138 ... yes
122/142 ... yes
122/143 ... yes
122/144 ... yes
122/145 ... yes
122/146 ... yes
35/148 ... yes
35/149 ... yes
166/150 ... yes
166/151 ... yes
166/152 ... yes
166/153 ... yes
166/154 ... yes
166/155 ... yes
166/156 ... yes
166/157 ... yes
166/158 ... yes
166/159 ... yes
166/160 ... yes
122/161 ... yes
122/162 ... yes
122/163 ... yes
122/165 ... yes
122/166 ... yes
122/167 ... yes
122/168 ... yes
122/169 ... yes
122/170 ... yes
122/171 ... yes
122/172 ... yes
122/173 ... yes
122/174 ... yes
122/175 ... yes
122/176 ... yes
122/177 ... yes
35/178 ... yes
122/179 ... yes
122/181 ... yes
122/182 ... yes
122/183 ... yes
122/184 ... yes
122/185 ... yes
122/186 ... yes
122/188 ... yes
122/189 ... yes
52/190 ... yes
35/191 ... yes
35/192 ... yes
125/196 ... yes
63/197 ... yes
124/198 ... yes
122/199 ... yes
122/200 ... yes
122/201 ... yes
122/202 ... yes
35/203 ... yes
122/204 ... yes
122/205 ... yes
35/206 ... yes
122/207 ... yes
122/208 ... yes
122/209 ... yes
122/210 ... yes
125/211 ... yes
122/213 ... yes
122/214 ... yes
122/215 ... yes
122/216 ... yes
122/217 ... yes
122/218 ... yes
122/219 ... yes
35/220 ... yes
122/221 ... yes
122/222 ... yes
122/225 ... yes
122/226 ... yes
122/227 ... yes
122/228 ... yes
415/229 ... yes
397/232 ... yes
122/234 ... yes
122/235 ... yes
35/236 ... yes
35/237 ... yes
35/238 ... yes
52/239 ... yes
332/240 ... yes
35/241 ... yes
35/242 ... yes
35/248 ... yes
38/250 ... yes
52/251 ... yes
122/252 ... yes
122/254 ... yes
122/255 ... yes
122/256 ... yes
122/257 ... yes
122/258 ... yes
79/259 ... yes
122/262 ... yes
122/263 ... yes
79/264 ... yes
435/265 ... yes
35/266 ... yes
122/267 ... yes
35/268 ... yes
397/269 ... yes
397/270 ... yes
35/271 ... yes
397/272 ... yes
122/273 ... yes
35/274 ... yes
122/275 ... yes
122/276 ... yes
35/277 ... yes
122/278 ... yes
497/279 ... yes
499/280 ... yes
501/281 ... yes
503/282 ... yes
122/283 ... yes
122/284 ... yes
515/285 ... yes
517/286 ... yes
519/287 ... yes
122/288 ... yes
122/289 ... yes
122/290 ... yes
525/291 ... yes
506/292 ... yes
35/293 ... yes
397/294 ... yes
122/296 ... yes
122/297 ... yes
122/298 ... yes
35/299 ... yes
35/300 ... yes
122/301 ... yes
35/302 ... yes
122/303 ... yes
122/304 ... yes
548/305 ... yes
550/306 ... yes
552/307 ... yes
562/308 ... yes
122/309 ... yes
525/310 ... yes
501/311 ... yes
35/312 ... yes
125/313 ... yes
575/314 ... yes
122/315 ... yes
35/319 ... yes
519/320 ... yes
386/323 ... yes
402/324 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 3.0.6 ? ... yes (3.1.4)
Git user has default SSH configuration? ... yes
Active users: ... 153
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

Things I tried:

1. Restarting the server it runs on
2. Restarting Gitlab several times
3. chmod -r 777 the entire gitlab-rails directory where uploads reside
4. Verifying the S3 bucket where we store packages is accessible (we have 10 other repos all working in the same Gitlab instance)
5. Fully updating Gitlab
6. gitlab reconfigure several times

typebug