Geo: Make geo:status rake task detect missing tracking database tables and suggest migration steps
What
When running gitlab-rake geo:status on a Geo secondary node, the command fails with the following error: ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "ci_secure_file_registry" does not exist LINE 10: WHERE a.attrelid = '"ci_secure_file_registry"'::regclass to due a missing table/incomplete migration.
This specific occurrence happened during the configuration of a new GitLab Dedicated instance here - https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/dedicated-migrations-group/dedicated-migrations/-/issues/45#note_2712545295
This error occurs because the Geo tracking database is missing required registry tables that were introduced in GitLab migrations. The ci_secure_file_registry table is typically the first to error because Geo::CiSecureFileReplicator is the first class in the REPLICATOR_CLASSES array, and the status command iterates through these classes in order.
This issue occurs when:
- The Geo secondary database has not been properly initialized with all required migrations
- Migrations were interrupted during the initial setup process
Workaround
-
Check the status of Geo migrations:
sudo gitlab-rake db:migrate:status:geo
Look for migrations showing as down, particularly:
CreateCiSecureFileRegistry
CreateContainerRepositoryRegistry
Other registry-related migrations- Run the Geo migrations if any of the above show as being "down":
sudo gitlab-rake db:migrate:geo- Verify the status command works:
sudo gitlab-rake geo:statusImplementation Plan
Improve gitlab-rake geo:status so it handles missing Geo tracking database registry tables gracefully.
- Before collecting Geo status for replicators, check whether the required Geo tracking database tables exist.
- If one or more registry tables are missing, do not allow the task to fail with a raw
ActiveRecord::StatementInvalid/PG::UndefinedTableexception. - Print an actionable error message that explains:
-
The Geo tracking database is missing required registry tables.
-
This can happen when Geo database migrations were not run or were interrupted.
-
The administrator should check migration status with:
sudo gitlab-rake db:migrate:status:geo -
If migrations are down, run:
sudo gitlab-rake db:migrate:geo
-
- Link to the Geo troubleshooting documentation section for this error.
- Add or update documentation in
doc/administration/geo/replication/troubleshooting/common.mdwith:- Symptom:
gitlab-rake geo:statusfails withPG::UndefinedTable. - Cause: missing Geo tracking database registry tables.
- Resolution: check and run Geo database migrations.
- Verification: rerun
sudo gitlab-rake geo:status.
- Symptom:
- Add tests for the rake task behavior when a required Geo tracking database table is missing, verifying that the task prints the actionable guidance instead of raising the raw database exception.