"No connection pool for 'ActiveRecord::Base' found." every 5 minutes after 18.3 -> 18.4 update

Summary

We have self-hosted GitLab environment installed using Omnibus on RHEL 9, where we experience intermittent 500 errors every ~5 minutes (sometimes more often, but in general every 5 minutes and it occurs for few seconds at a time), with following exceptions visible in exceptions_log.json (see below).

The GitLab instance uses external PostgreSQL 16 cluster and gitlab database and this issue didn't occur until we've updated to 18.4.1 yesterday (I've downgraded the instance to 18.4.0 to see whether it wasn't anything related to the bugfix version, but the issue is still present there - when we downgrade to 18.3.3, it works fine).

What I can see is various stacktraces and from nginx logs it's seems like just random endpoints (see logs below).

The instance really just has

gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_database'] = 'gitlab'

gitlab_rails['db_host'] = '<ip>'
gitlab_rails['db_port'] = '5432'
gitlab_rails['db_username'] = 'gitlab'
gitlab_rails['db_password'] = '<pass>'

configured for the database, so I assume this might be due to some dependency update or something internal about the connection handling - possibly some timeout/threshold is reached so often that causes the connection not to be available. However Googling that yielded nothing helpful.

In tcpdump there are no visible TCP RSTs or any visible issues where PostgreSQL would not be accessible (and I've verified it's actually available during that time).

As I've tried all I can think of, would anyone have any insights into what to check or what could be the underlying issue?

Thanks

Steps to reproduce

n/a

Example Project

What is the current bug behavior?

GitLab raises HTTP 500 errors with "ConnectionNotEstablished" error in logs every few minutes.

What is the expected correct behavior?

GitLab works fine and does not raise any errors.

Relevant logs and/or screenshots

{
  "severity": "ERROR",
  "time": "2025-10-01T17:00:14.761Z",
  "correlation_id": "01K6GBFA0WYFNKR5JHVQH5HHMZ",
  "exception.class": "ActiveRecord::ConnectionNotEstablished",
  "exception.message": "No connection pool for 'ActiveRecord::Base' found.",
  "exception.backtrace": [
    "lib/feature.rb:24:in `connection'",
    "lib/gitlab/database/query_analyzers/log_large_in_lists.rb:24:in `enabled?'",
    "lib/gitlab/database/query_analyzer.rb:95:in `block in begin!'",
    "lib/gitlab/database/query_analyzer.rb:92:in `select'",
    "lib/gitlab/database/query_analyzer.rb:92:in `begin!'",
    "lib/gitlab/database/query_analyzer.rb:80:in `within'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
    "lib/ci/job_token/middleware.rb:11:in `call'",
    "lib/gitlab/middleware/multipart.rb:174:in `call'",
    "lib/gitlab/middleware/read_only/controller.rb:40:in `call'",
    "lib/gitlab/middleware/read_only.rb:18:in `call'",
    "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
    "lib/gitlab/middleware/secure_headers.rb:11:in `call'",
    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
    "lib/gitlab/middleware/path_traversal_check.rb:40:in `call'",
    "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
    "lib/gitlab/middleware/json_validation.rb:179:in `allow_if_validated'",
    "lib/gitlab/middleware/json_validation.rb:160:in `call'",
    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
    "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
    "lib/gitlab/middleware/request_context.rb:15:in `call'",
    "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
    "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
    "lib/gitlab/middleware/compressed_json.rb:44:in `call'",
    "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
    "lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'",
    "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
    "lib/gitlab/middleware/release_env.rb:12:in `call'"
  ],
  "user.username": null,
  "tags.program": "web",
  "tags.locale": "en",
  "tags.feature_category": null,
  "tags.correlation_id": "01K6GBFA0WYFNKR5JHVQH5HHMZ"
}
{
  "severity": "ERROR",
  "time": "2025-10-01T17:00:14.772Z",
  "correlation_id": "01K6GBFA0WYFNKR5JHVQH5HHMZ",
  "meta.caller_id": "POST /api/:version/internal/allowed",
  "meta.remote_ip": "127.0.0.1",
  "meta.feature_category": "source_code_management",
  "meta.client_id": "ip/127.0.0.1",
  "exception.class": "ActiveRecord::ConnectionNotEstablished",
  "exception.message": "ActiveRecord::ConnectionNotEstablished",
  "exception.backtrace": [
    "lib/gitlab/database/load_balancing/configuration.rb:66:in `db_config_name'",
    "lib/gitlab/database/load_balancing/load_balancer.rb:38:in `name'",
    "lib/gitlab/database/load_balancing/session_map.rb:51:in `lookup'",
    "lib/gitlab/database/load_balancing/session_map.rb:13:in `current'",
    "lib/gitlab/database/load_balancing/connection_proxy.rb:134:in `current_session'",
    "lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `read_using_load_balancer'",
    "lib/gitlab/database/load_balancing/connection_proxy.rb:54:in `block (2 levels) in <class:ConnectionProxy>'",
    "config/initializers/postgresql_cte.rb:107:in `build_arel'",
    "lib/api/support/git_access_actor.rb:19:in `from_params'",
    "lib/api/helpers/internal_helpers.rb:13:in `actor'",
    "lib/api/internal/base.rb:19:in `block in <class:Base>'",
    "lib/gitlab/middleware/ip_address.rb:11:in `call'",
    "lib/api/api_guard.rb:266:in `call'",
    "config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes'",
    "config/initializers/action_dispatch_journey_router.rb:25:in `map!'",
    "config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'",
    "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
    "lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
    "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
    "lib/gitlab/middleware/memory_report.rb:13:in `call'",
    "lib/gitlab/middleware/speedscope.rb:13:in `call'",
    "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
    "lib/gitlab/middleware/go.rb:21:in `call'",
    "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
    "lib/gitlab/database/query_analyzer.rb:83:in `within'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
    "lib/ci/job_token/middleware.rb:11:in `call'",
    "lib/gitlab/middleware/multipart.rb:174:in `call'",
    "lib/gitlab/middleware/read_only/controller.rb:40:in `call'",
    "lib/gitlab/middleware/read_only.rb:18:in `call'",
    "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
    "lib/gitlab/middleware/secure_headers.rb:11:in `call'",
    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
    "lib/gitlab/middleware/path_traversal_check.rb:40:in `call'",
    "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
    "lib/gitlab/middleware/json_validation.rb:179:in `allow_if_validated'",
    "lib/gitlab/middleware/json_validation.rb:160:in `call'",
    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
    "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
    "lib/gitlab/middleware/request_context.rb:15:in `call'",
    "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
    "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
    "lib/gitlab/middleware/compressed_json.rb:44:in `call'",
    "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
    "lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'",
    "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
    "lib/gitlab/middleware/release_env.rb:12:in `call'"
  ],
  "user.username": null,
  "tags.program": "web",
  "tags.locale": "en",
  "tags.feature_category": "source_code_management",
  "tags.correlation_id": "01K6GBFA0WYFNKR5JHVQH5HHMZ"
}

Database diagnostics (in UI -> Admin -> Monitoring -> Database diagnostics) shows everything okay:

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	3.2.8
Gem Version:	3.7.1
Bundler Version:2.7.1
Rake Version:	13.0.6
Redis Version:	7.2.10
Sidekiq Version:7.3.9
Go Version:	unknown

GitLab information
Version:	18.4.1
Revision:	e88a8d1d1c2
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	16.8
URL:		https://domain
HTTP Clone URL:	https://domain/some-group/some-project.git
SSH Clone URL:	git@domain:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: saml

GitLab Shell
Version:	14.45.2
Repository storages:
- default: 	tcp://ip:8075
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	tcp://ip:8075
- default Version: 	18.4.1
- default Git Version: 	2.50.1

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.45.2 ? ... OK (14.45.2) Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK cluster ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... Redis version >= 6.2.14? ... yes Ruby version >= 3.0.6 ? ... yes (3.2.8) Git user has default SSH configuration? ... yes Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

n/a

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited Oct 02, 2025 by 🤖 GitLab Bot 🤖