Geo: Tracking database connection check fails with AWS RDS despite healthy connection

Summary

When running gitlab-rake gitlab:geo:check, it returns:

GitLab Geo tracking database is correctly configured ... no
Check the tracking database configuration as the connection could not be established

However, the actual database connection appears to be working properly and the Geo sync status is healthy.

This seems to happen because the call to ::Geo::TrackingBase.connection.active? returns false(code here). However, Geo::TrackingBase.connection.execute("SELECT 1") still works, and calling Geo::TrackingBase.connection.reconnect! makes ::Geo::TrackingBase.connection.active? return true again.

We also noticed that .active? method on the main db connection object also returns false .

Steps to reproduce

  • Use AWS RDS as the database for tracking database for Geo secondary.
  • Run ::Geo::TrackingBase.connection.active? from the Rails console.

Example Project

We noticed this issue while working on this self-managed ticket: https://gitlab.zendesk.com/agent/tickets/638509.

What is the current bug behavior?

gitlab-rake gitlab:geo:check reports that a connection to the tracking database could not be established, even though the connection appears to be intact.

What is the expected correct behavior?

gitlab-rake gitlab:geo:check should report the correct connection status.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

 (For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:env:info\`)  (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\`)

Results of GitLab application Check

Expand for output related to the GitLab application check 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:check SANITIZE=true`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`)

(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.