### Summary `timeout_helpers_spec.rb` leaves the main connection with no statement timeout. A later spec, `diagnostics_rake_spec.rb`, reports that as an error and exits with a transaction still open. In `single-db-ci-connection` jobs the `ci` connection shares that database, so it blocks on that transaction and the examples after it fail. - Job: [rspec unit pg17 single-db-ci-connection 19/35](https://gitlab.com/gitlab-org/gitlab/-/jobs/17185888351), a retry of [17181637258](https://gitlab.com/gitlab-org/gitlab/-/jobs/17181637258), on https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253800 (it touches only `ee/` Geo files). - Both runs fail the same 22 examples: `diagnostics_rake_spec.rb:100`, 16 in `spec/workers/ci/pipeline_creation_metrics_worker_spec.rb` (`PG::InFailedSqlTransaction`) and 5 in `spec/services/import/github/cancel_project_import_service_spec.rb` (`PG::QueryCanceled`). - Both also report 2 errors outside examples: the cleaner `DELETE` times out (`spec/support/db_cleaner.rb:168`) in the `after(:all)` hook at `spec/support/rake.rb:18`, and `spec/support/database/auto_explain.rb:21` raises `PG::QueryCanceled` in the `after(:suite)` hook. ### Cause 1. The `after` hooks at `timeout_helpers_spec.rb:31`, `:102` and `:129` run `RESET statement_timeout`. That returns to the server default (0), not the 15s that `database.yml` sets. In the reproduction, a probe right after it showed `main` at 0 and `ci` at 15s. 2. `diagnostics_rake_spec.rb:100` runs the real checks. The timeouts check added in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252148 reports an unlimited `statement_timeout` as an error, and the rake task exits with `SystemExit`. 3. The `main` connection is then left idle in a transaction, its last statement the cleaner's `ALTER TABLE ... ENABLE ALWAYS TRIGGER` (`spec/support/db_cleaner.rb:225`). 4. In this mode the `ci` connection is a second backend on the same database. Its cleaner `DELETE` blocks on that transaction until the 15s timeout, and later examples fail. ```mermaid flowchart TD A["timeout_helpers_spec.rb:31<br/>RESET statement_timeout"] --> B["main connection<br/>statement_timeout is 0"] B --> C["diagnostics_rake_spec.rb:100<br/>statement_timeout_unlimited error"] C --> D["diagnostics.rake<br/>SystemExit"] D --> E["main connection<br/>idle in transaction, holding locks"] E --> F["ci connection, same database<br/>cleaner DELETE blocks"] F --> G["15s timeout: PG::QueryCanceled"] G --> H["later examples:<br/>InFailedSqlTransaction or QueryCanceled"] ``` I don't know why the `main` transaction stays open after the `SystemExit`. In the CI job, `timeout_helpers_spec.rb` ran about 12 minutes earlier (10 in the first run) in the same RSpec process. ### Steps to reproduce On a GitLab source checkout at master `1d5bd42d58069fea6abf0da73275e04e9ee6dec2`, with `database.yml` built as `setup_database_yml` in `scripts/utils.sh` does for `single-db-ci-connection` (`ci` pointing at the same database, `statement_timeout` 15s), run RSpec with `--order defined`: 1. The three failing files alone: 32 examples, 0 failures. 1. `timeout_helpers_spec.rb` first, then the three: 42 examples, the same 22 failures and 1 error outside examples. With `CI=true` the `auto_explain.rb` error makes 2, as in CI. 1. `timeout_helpers_spec.rb` and `diagnostics_rake_spec.rb` alone: 1 failure and 1 error outside examples. 1. The same four files with decomposed databases: only `diagnostics_rake_spec.rb:100` fails, with no hang and no cascade. Master with and without the diff of that MR gives identical results. ### Proposed Resolution Two directions for the owners, not exclusive: - `timeout_helpers_spec.rb`: restore the configured timeout in those `after` hooks instead of `RESET`. - `diagnostics_rake_spec.rb:100`: isolate the real-database example, so a failing check cannot leave a transaction open on the shared connection. I haven't established which is right. An AI coding assistant ran these reproductions on a GitLab source checkout on a test VM and drafted this report. ### Logs ``` Failures: 1) gitlab:db:diagnostics renders a report for the real database Failure/Error: raise SystemExitDetected, "SystemExit should be rescued in the tests!" # ./spec/support/system_exit_detected.rb:16:in `rescue in block (2 levels) in <top (required)>' # --- Caused by: --- # SystemExit: # Database diagnostics found errors. Review the output above. # ./lib/tasks/gitlab/db/diagnostics.rake:9:in `block (3 levels) in <top (required)>' 2) Ci::PipelineCreationMetricsWorker#perform when pipeline exists when pipeline has builds tracks build creation events ActiveRecord::StatementInvalid: PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block An error occurred in an `after(:suite)` hook. ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout CONTEXT: while inserting index tuple (0,13) in relation "pg_extension_name_index" # ./spec/support/database/auto_explain.rb:21:in `block in record' ```