Geo: Make geo:status rake task detect missing tracking database tables and suggest migration steps

What

When running gitlab-rake geo:status on a Geo secondary node, the command fails with the following error: ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "ci_secure_file_registry" does not exist LINE 10: WHERE a.attrelid = '"ci_secure_file_registry"'::regclass to due a missing table/incomplete migration.

This specific occurrence happened during the configuration of a new GitLab Dedicated instance here - https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/dedicated-migrations-group/dedicated-migrations/-/issues/45#note_2712545295

This error occurs because the Geo tracking database is missing required registry tables that were introduced in GitLab migrations. The ci_secure_file_registry table is typically the first to error because Geo::CiSecureFileReplicator is the first class in the REPLICATOR_CLASSES array, and the status command iterates through these classes in order.

This issue occurs when:

  • The Geo secondary database has not been properly initialized with all required migrations
  • Migrations were interrupted during the initial setup process

Workaround

  1. Check the status of Geo migrations:

    sudo gitlab-rake db:migrate:status:geo
    

Look for migrations showing as down, particularly:

CreateCiSecureFileRegistry
CreateContainerRepositoryRegistry
Other registry-related migrations
  1. Run the Geo migrations if any of the above show as being "down":
sudo gitlab-rake db:migrate:geo
  1. Verify the status command works:
sudo gitlab-rake geo:status

Implementation Plan

Improve gitlab-rake geo:status so it handles missing Geo tracking database registry tables gracefully.

  1. Before collecting Geo status for replicators, check whether the required Geo tracking database tables exist.
  2. If one or more registry tables are missing, do not allow the task to fail with a raw ActiveRecord::StatementInvalid / PG::UndefinedTable exception.
  3. Print an actionable error message that explains:
    • The Geo tracking database is missing required registry tables.

    • This can happen when Geo database migrations were not run or were interrupted.

    • The administrator should check migration status with:

      sudo gitlab-rake db:migrate:status:geo
    • If migrations are down, run:

      sudo gitlab-rake db:migrate:geo
  4. Link to the Geo troubleshooting documentation section for this error.
  5. Add or update documentation in doc/administration/geo/replication/troubleshooting/common.md with:
    • Symptom: gitlab-rake geo:status fails with PG::UndefinedTable.
    • Cause: missing Geo tracking database registry tables.
    • Resolution: check and run Geo database migrations.
    • Verification: rerun sudo gitlab-rake geo:status.
  6. Add tests for the rake task behavior when a required Geo tracking database table is missing, verifying that the task prints the actionable guidance instead of raising the raw database exception.
Edited by 🤖 GitLab Bot 🤖