You need to sign in or sign up before continuing.
NoMethodError: undefined method `include?' for nil:NilClass in ip_rate_limiter.rb
We received a report of a persistent HTTP 500 response from someone accessing our self-hosted Omnibus instance https://www.drupal.org/project/infrastructure/issues/3543850
Summary
This happened on git push with an https://… remote using an access token. I believe the backtrace below should be helpful.
Steps to reproduce
The last few lines of the backtrace involve ip_rate_limiter.rb, so I imagine there must be some rate limiting involved somehow. The original reporter mentioned their “collegues have the same issue,” so there is some likelihood of traffic from a few people coming through a single IP.
Example Project
n/a
What is the current bug behavior?
They receive an error like
fatal: unable to access 'https://git.drupalcode.org/issue/pathalias_extend-3543394.git/': The requested URL returned error: 500What is the expected correct behavior?
Either a success or an informative error about rate limiting.
Relevant logs and/or screenshots
This stack trace lands in production_json.log
{
"action" : "info_refs",
"controller" : "Repositories::GitHttpController",
"correlation_id" : "01K3V6S15E9B305CFY6QCCRABF",
"cpu_s" : 0.01472,
"db_cached_count" : 0,
"db_ci_cached_count" : 0,
"db_ci_count" : 0,
"db_ci_duration_s" : 0,
"db_ci_replica_cached_count" : 0,
"db_ci_replica_count" : 0,
"db_ci_replica_duration_s" : 0,
"db_ci_replica_txn_count" : 0,
"db_ci_replica_txn_duration_s" : 0,
"db_ci_replica_txn_max_duration_s" : 0,
"db_ci_replica_wal_cached_count" : 0,
"db_ci_replica_wal_count" : 0,
"db_ci_replica_write_count" : 0,
"db_ci_txn_count" : 0,
"db_ci_txn_duration_s" : 0,
"db_ci_txn_max_duration_s" : 0,
"db_ci_wal_cached_count" : 0,
"db_ci_wal_count" : 0,
"db_ci_write_count" : 0,
"db_count" : 8,
"db_duration_s" : 0.00078,
"db_main_cached_count" : 0,
"db_main_count" : 8,
"db_main_duration_s" : 0.002,
"db_main_replica_cached_count" : 0,
"db_main_replica_count" : 0,
"db_main_replica_duration_s" : 0,
"db_main_replica_txn_count" : 0,
"db_main_replica_txn_duration_s" : 0,
"db_main_replica_txn_max_duration_s" : 0,
"db_main_replica_wal_cached_count" : 0,
"db_main_replica_wal_count" : 0,
"db_main_replica_write_count" : 0,
"db_main_txn_count" : 0,
"db_main_txn_duration_s" : 0,
"db_main_txn_max_duration_s" : 0,
"db_main_wal_cached_count" : 0,
"db_main_wal_count" : 0,
"db_main_write_count" : 0,
"db_primary_cached_count" : 0,
"db_primary_count" : 8,
"db_primary_duration_s" : 0.002,
"db_primary_txn_count" : 0,
"db_primary_txn_duration_s" : 0,
"db_primary_txn_max_duration_s" : 0,
"db_primary_wal_cached_count" : 0,
"db_primary_wal_count" : 0,
"db_primary_write_count" : 0,
"db_replica_cached_count" : 0,
"db_replica_count" : 0,
"db_replica_duration_s" : 0,
"db_replica_txn_count" : 0,
"db_replica_txn_duration_s" : 0,
"db_replica_txn_max_duration_s" : 0,
"db_replica_wal_cached_count" : 0,
"db_replica_wal_count" : 0,
"db_replica_write_count" : 0,
"db_txn_count" : 0,
"db_write_count" : 0,
"duration_s" : 0.00534,
"exception.backtrace" : [
"lib/gitlab/auth/ip_rate_limiter.rb:37:in `block in trusted_ip?'",
"lib/gitlab/auth/ip_rate_limiter.rb:37:in `any?'",
"lib/gitlab/auth/ip_rate_limiter.rb:37:in `trusted_ip?'",
"lib/gitlab/auth/ip_rate_limiter.rb:43:in `skip_rate_limit?'",
"lib/gitlab/auth/ip_rate_limiter.rb:31:in `banned?'",
"lib/gitlab/auth.rb:122:in `find_for_git_client'",
"app/controllers/repositories/git_http_client_controller.rb:137:in `handle_basic_authentication'",
"app/controllers/repositories/git_http_client_controller.rb:55:in `authenticate_user'",
"ee/app/controllers/ee/repositories/git_http_controller.rb:128:in `authenticate_user'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:403:in `block in make_lambda'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:202:in `block (2 levels) in halting'",
"actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:203:in `block in halting'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `block in invoke_before'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `each'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `invoke_before'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:119:in `block in run_callbacks'",
"lib/gitlab/ip_address_state.rb:11:in `with'",
"app/controllers/application_controller.rb:462:in `set_current_ip_address'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:34:in `block in sentry_around_action'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:138:in `with_child_span'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:515:in `with_child_span'",
"sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:141:in `run_callbacks'",
"actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:258:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/rescue.rb:25:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:74:in `block in process_action'",
"activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `block in instrument'",
"activesupport (7.1.5.1) lib/active_support/notifications/instrumenter.rb:58:in `instrument'",
"activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `instrument'",
"actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:73:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/params_wrapper.rb:261:in `process_action'",
"activerecord (7.1.5.1) lib/active_record/railties/controller_runtime.rb:32:in `process_action'",
"actionpack (7.1.5.1) lib/abstract_controller/base.rb:160:in `process'",
"actionview (7.1.5.1) lib/action_view/rendering.rb:40:in `process'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:227:in `dispatch'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `block in dispatch'",
"lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `dispatch'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:32:in `serve'",
"actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:51:in `block in serve'",
"config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes'",
"config/initializers/action_dispatch_journey_router.rb:25:in `map!'",
"config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'",
"actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:32:in `serve'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:882:in `call'",
"gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'",
"omniauth (2.1.3) lib/omniauth/strategy.rb:202:in `call!'",
"omniauth (2.1.3) lib/omniauth/strategy.rb:169:in `call'",
"flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'",
"flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
"lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/go.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:83:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/ci/job_token/middleware.rb:11:in `call'",
"batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'",
"rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'",
"apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'",
"warden (1.2.9) lib/warden/manager.rb:36:in `block in call'",
"warden (1.2.9) lib/warden/manager.rb:34:in `catch'",
"warden (1.2.9) lib/warden/manager.rb:34:in `call'",
"rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'",
"rack (2.2.17) lib/rack/tempfile_reaper.rb:15:in `call'",
"rack (2.2.17) lib/rack/etag.rb:27:in `call'",
"rack (2.2.17) lib/rack/conditional_get.rb:27:in `call'",
"rack (2.2.17) lib/rack/head.rb:12:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/http/permissions_policy.rb:36:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/http/content_security_policy.rb:36:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:40:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
"rack (2.2.17) lib/rack/session/abstract/id.rb:266:in `context'",
"rack (2.2.17) lib/rack/session/abstract/id.rb:260:in `call'",
"lib/gitlab/middleware/secure_headers.rb:11:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/cookies.rb:689:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:101:in `run_callbacks'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:28:in `call'",
"sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'",
"lib/gitlab/middleware/path_traversal_check.rb:40:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'",
"railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `block in call'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `block in tagged'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:39:in `tagged'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `tagged'",
"activesupport (7.1.5.1) lib/active_support/broadcast_logger.rb:241:in `method_missing'",
"railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/remote_ip.rb:92:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:15:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"request_store (1.7.0) lib/request_store/middleware.rb:19:in `call'",
"rack (2.2.17) lib/rack/method_override.rb:24:in `call'",
"rack (2.2.17) lib/rack/runtime.rb:22:in `call'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'",
"rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:44:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/executor.rb:14:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'",
"gitlab-labkit (0.39.0) lib/labkit/middleware/rack.rb:22:in `block in call'",
"gitlab-labkit (0.39.0) lib/labkit/context.rb:35:in `with_context'",
"gitlab-labkit (0.39.0) lib/labkit/middleware/rack.rb:21:in `call'",
"rack (2.2.17) lib/rack/sendfile.rb:110:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/request_id.rb:28:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"railties (7.1.5.1) lib/rails/engine.rb:536:in `call'",
"railties (7.1.5.1) lib/rails/railtie.rb:226:in `public_send'",
"railties (7.1.5.1) lib/rails/railtie.rb:226:in `method_missing'",
"lib/gitlab/middleware/release_env.rb:12:in `call'",
"rack (2.2.17) lib/rack/urlmap.rb:74:in `block in call'",
"rack (2.2.17) lib/rack/urlmap.rb:58:in `each'",
"rack (2.2.17) lib/rack/urlmap.rb:58:in `call'",
"puma (6.6.1) lib/puma/configuration.rb:279:in `call'",
"puma (6.6.1) lib/puma/request.rb:99:in `block in handle_request'",
"puma (6.6.1) lib/puma/thread_pool.rb:390:in `with_force_shutdown'",
"puma (6.6.1) lib/puma/request.rb:98:in `handle_request'",
"puma (6.6.1) lib/puma/server.rb:472:in `process_client'",
"puma (6.6.1) lib/puma/server.rb:254:in `block in run'",
"puma (6.6.1) lib/puma/thread_pool.rb:167:in `block in spawn_thread'"
],
"exception.class" : "NoMethodError",
"exception.message" : "undefined method `include?' for nil:NilClass",
"format" : "*/*",
"mem_bytes" : 705120,
"mem_mallocs" : 1511,
"mem_objects" : 9847,
"mem_total_bytes" : 1099000,
"meta.caller_id" : "Repositories::GitHttpController#info_refs",
"meta.feature_category" : "source_code_management",
"method" : "GET",
"params" : [
{
"key" : "service",
"value" : "git-receive-pack"
},
{
"key" : "repository_path",
"value" : "issue/pathalias_extend-3543394.git"
}
],
"path" : "/issue/pathalias_extend-3543394.git/info/refs",
"path_traversal_check_duration_s" : 0.000198,
"pid" : 2154212,
"rate_limiting_gates" : [],
"redis_calls" : 1,
"redis_duration_s" : 8.8e-05,
"redis_rate_limiting_calls" : 1,
"redis_rate_limiting_duration_s" : 8.8e-05,
"redis_rate_limiting_read_bytes" : 1,
"redis_rate_limiting_write_bytes" : 74,
"redis_read_bytes" : 1,
"redis_write_bytes" : 74,
"remote_ip" : "95.90.240.254",
"repository_storage" : "default",
"request_urgency" : "default",
"status" : 500,
"target_duration_s" : 1,
"time" : "2025-08-29T15:22:43.775Z",
"ua" : "git/2.37.2.windows.2",
"view_duration_s" : 0,
"worker_id" : "puma_12"
}Output of checks
n/a
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 3.2.8 Gem Version: 3.6.9 Bundler Version:2.7.1 Rake Version: 13.0.6 Redis Version: 7.2.10 Sidekiq Version:7.3.9 Go Version: unknown GitLab information Version: 18.3.1-ee Revision: cb15859792b Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 16.8 URL: https://git.drupalcode.org HTTP Clone URL: https://git.drupalcode.org/some-group/some-project.git SSH Clone URL: git@git.drupal.org:some-group/some-project.git Elasticsearch: yes Geo: yes Geo node: Primary Using LDAP: no Using Omniauth: yes Omniauth Providers: jwt GitLab Shell Version: 14.44.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 18.3.1 - default Git Version: 2.50.1.gl1
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 14.44.0 ? ... OK (14.44.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/2 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/4 ... yes 2/6 ... yes [ 175938 yesses trimmmed] 49196/190768 ... yes 49196/190769 ... yes Redis version >= 6.2.14? ... yes Ruby version >= 3.0.6 ? ... yes (3.2.8) Git user has default SSH configuration? ... yes Active users: ... 193237 Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-9.x or OpenSearch version 1.x-3.x ... yes (elasticsearch 7.10.2) All migrations must be finished before doing a major upgrade ... no (You have 11 pending migrations.) For more information see: https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html#all-migrations-must-be-finished-before-doing-a-major-upgrade Try fixing it: Wait for all advanced search migrations to complete. To list pending migrations, run `sudo gitlab-rake gitlab:elastic:list_pending_migrations` Checking GitLab App ... Finished Checking Geo ... GitLab Geo is available ... GitLab Geo is enabled ... yes This machine's Geo node name matches a database record ... yes, found a primary node named "https://git.drupalcode.org/" GitLab Geo tracking database is not configured after promotion ... yes HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... yes Git user has default SSH configuration? ... yes OpenSSH configured to use AuthorizedKeysCommand ... skipped Reason: Cannot access OpenSSH configuration file Try fixing it: This is expected if you are using SELinux. You may want to check configuration manually For more information see: doc/administration/operations/fast_ssh_key_lookup.md GitLab configured to disable writing to authorized_keys file ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking Geo ... Finished Checking GitLab subtasks ... Finished
Possible fixes
Patch release information for backports
If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.
Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.
High-severity bug remediation
To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.
Edited by 🤖 GitLab Bot 🤖