"No connection pool for 'ActiveRecord::Base' found." every 5 minutes after 18.3 -> 18.4 update (#573581) · Issues · GitLab.org / GitLab

"No connection pool for 'ActiveRecord::Base' found." every 5 minutes after 18.3 -> 18.4 update

### Summary  We have self-hosted GitLab environment installed using Omnibus on RHEL 9, where we experience intermittent 500 errors every ~5 minutes (sometimes more often, but in general every 5 minutes and it occurs for few seconds at a time), with following exceptions visible in exceptions_log.json (see below). The GitLab instance uses external PostgreSQL 16 cluster and `gitlab` database and this issue didn't occur until we've updated to 18.4.1 yesterday (I've downgraded the instance to 18.4.0 to see whether it wasn't anything related to the bugfix version, but the issue is still present there - when we downgrade to 18.3.3, it works fine). What I can see is various stacktraces and from nginx logs it's seems like just random endpoints (see logs below). The instance really just has ``` gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'utf8' gitlab_rails['db_database'] = 'gitlab' gitlab_rails['db_host'] = '<ip>' gitlab_rails['db_port'] = '5432' gitlab_rails['db_username'] = 'gitlab' gitlab_rails['db_password'] = '<pass>' ``` configured for the database, so I assume this might be due to some dependency update or something internal about the connection handling - possibly some timeout/threshold is reached so often that causes the connection not to be available. However Googling that yielded nothing helpful. In tcpdump there are no visible TCP RSTs or any visible issues where PostgreSQL would not be accessible (and I've verified it's actually available during that time). As I've tried all I can think of, would anyone have any insights into what to check or what could be the underlying issue? Thanks ### Steps to reproduce  n/a ### Example Project  ### What is the current *bug* behavior?  GitLab raises HTTP 500 errors with "ConnectionNotEstablished" error in logs every few minutes. ### What is the expected *correct* behavior?  GitLab works fine and does not raise any errors. ### Relevant logs and/or screenshots  ``` { "severity": "ERROR", "time": "2025-10-01T17:00:14.761Z", "correlation_id": "01K6GBFA0WYFNKR5JHVQH5HHMZ", "exception.class": "ActiveRecord::ConnectionNotEstablished", "exception.message": "No connection pool for 'ActiveRecord::Base' found.", "exception.backtrace": [ "lib/feature.rb:24:in `connection'", "lib/gitlab/database/query_analyzers/log_large_in_lists.rb:24:in `enabled?'", "lib/gitlab/database/query_analyzer.rb:95:in `block in begin!'", "lib/gitlab/database/query_analyzer.rb:92:in `select'", "lib/gitlab/database/query_analyzer.rb:92:in `begin!'", "lib/gitlab/database/query_analyzer.rb:80:in `within'", "lib/gitlab/middleware/query_analyzer.rb:11:in `call'", "lib/ci/job_token/middleware.rb:11:in `call'", "lib/gitlab/middleware/multipart.rb:174:in `call'", "lib/gitlab/middleware/read_only/controller.rb:40:in `call'", "lib/gitlab/middleware/read_only.rb:18:in `call'", "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'", "lib/gitlab/middleware/secure_headers.rb:11:in `call'", "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", "lib/gitlab/middleware/path_traversal_check.rb:40:in `call'", "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'", "lib/gitlab/middleware/json_validation.rb:179:in `allow_if_validated'", "lib/gitlab/middleware/json_validation.rb:160:in `call'", "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", "lib/gitlab/middleware/request_context.rb:15:in `call'", "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'", "config/initializers/fix_local_cache_middleware.rb:11:in `call'", "lib/gitlab/middleware/compressed_json.rb:44:in `call'", "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'", "lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'", "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'", "lib/gitlab/middleware/release_env.rb:12:in `call'" ], "user.username": null, "tags.program": "web", "tags.locale": "en", "tags.feature_category": null, "tags.correlation_id": "01K6GBFA0WYFNKR5JHVQH5HHMZ" } { "severity": "ERROR", "time": "2025-10-01T17:00:14.772Z", "correlation_id": "01K6GBFA0WYFNKR5JHVQH5HHMZ", "meta.caller_id": "POST /api/:version/internal/allowed", "meta.remote_ip": "127.0.0.1", "meta.feature_category": "source_code_management", "meta.client_id": "ip/127.0.0.1", "exception.class": "ActiveRecord::ConnectionNotEstablished", "exception.message": "ActiveRecord::ConnectionNotEstablished", "exception.backtrace": [ "lib/gitlab/database/load_balancing/configuration.rb:66:in `db_config_name'", "lib/gitlab/database/load_balancing/load_balancer.rb:38:in `name'", "lib/gitlab/database/load_balancing/session_map.rb:51:in `lookup'", "lib/gitlab/database/load_balancing/session_map.rb:13:in `current'", "lib/gitlab/database/load_balancing/connection_proxy.rb:134:in `current_session'", "lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `read_using_load_balancer'", "lib/gitlab/database/load_balancing/connection_proxy.rb:54:in `block (2 levels) in <class:ConnectionProxy>'", "config/initializers/postgresql_cte.rb:107:in `build_arel'", "lib/api/support/git_access_actor.rb:19:in `from_params'", "lib/api/helpers/internal_helpers.rb:13:in `actor'", "lib/api/internal/base.rb:19:in `block in <class:Base>'", "lib/gitlab/middleware/ip_address.rb:11:in `call'", "lib/api/api_guard.rb:266:in `call'", "config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes'", "config/initializers/action_dispatch_journey_router.rb:25:in `map!'", "config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'", "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'", "lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'", "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'", "lib/gitlab/middleware/memory_report.rb:13:in `call'", "lib/gitlab/middleware/speedscope.rb:13:in `call'", "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'", "lib/gitlab/middleware/go.rb:21:in `call'", "lib/gitlab/etag_caching/middleware.rb:21:in `call'", "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'", "lib/gitlab/database/query_analyzer.rb:83:in `within'", "lib/gitlab/middleware/query_analyzer.rb:11:in `call'", "lib/ci/job_token/middleware.rb:11:in `call'", "lib/gitlab/middleware/multipart.rb:174:in `call'", "lib/gitlab/middleware/read_only/controller.rb:40:in `call'", "lib/gitlab/middleware/read_only.rb:18:in `call'", "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'", "lib/gitlab/middleware/secure_headers.rb:11:in `call'", "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", "lib/gitlab/middleware/path_traversal_check.rb:40:in `call'", "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'", "lib/gitlab/middleware/json_validation.rb:179:in `allow_if_validated'", "lib/gitlab/middleware/json_validation.rb:160:in `call'", "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", "lib/gitlab/middleware/request_context.rb:15:in `call'", "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'", "config/initializers/fix_local_cache_middleware.rb:11:in `call'", "lib/gitlab/middleware/compressed_json.rb:44:in `call'", "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'", "lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'", "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'", "lib/gitlab/middleware/release_env.rb:12:in `call'" ], "user.username": null, "tags.program": "web", "tags.locale": "en", "tags.feature_category": "source_code_management", "tags.correlation_id": "01K6GBFA0WYFNKR5JHVQH5HHMZ" } ``` Database diagnostics (in UI -> Admin -> Monitoring -> Database diagnostics) shows everything okay: ![image](/uploads/8e351628dd1031e46b6087f4d4585154/image.png) ### Output of checks      #### Results of GitLab environment info  <details> <summary>Expand for output related to GitLab environment info</summary> <pre> System information System: Current User: git Using RVM: no Ruby Version: 3.2.8 Gem Version: 3.7.1 Bundler Version:2.7.1 Rake Version: 13.0.6 Redis Version: 7.2.10 Sidekiq Version:7.3.9 Go Version: unknown GitLab information Version: 18.4.1 Revision: e88a8d1d1c2 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 16.8 URL: https://domain HTTP Clone URL: https://domain/some-group/some-project.git SSH Clone URL: git@domain:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 14.45.2 Repository storages: - default: tcp://ip:8075 GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: tcp://ip:8075 - default Version: 18.4.1 - default Git Version: 2.50.1 </pre> </details> #### Results of GitLab application Check  <details> <summary>Expand for output related to the GitLab application check</summary> <pre> Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 14.45.2 ? ... OK (14.45.2) Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK cluster ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... <all yes> Redis version >= 6.2.14? ... yes Ruby version >= 3.0.6 ? ... yes (3.2.8) Git user has default SSH configuration? ... yes Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished </pre> </details> ### Possible fixes  n/a ### Patch release information for backports If the bug fix needs to be backported in a [patch release](https://handbook.gitlab.com/handbook/engineering/releases/patch-releases) to a version under [the maintenance policy](https://docs.gitlab.com/policy/maintenance/), please follow the steps on the [patch release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md). Refer to the [internal "Release Information" dashboard](https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1) for information about the next patch release, including the targeted versions, expected release date, and current status. #### High-severity bug remediation To remediate high-severity issues requiring an [internal release](https://handbook.gitlab.com/handbook/engineering/releases/internal-releases/) for single-tenant SaaS instances, refer to the [internal release process for engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md?ref_type=heads).

issue