Geo: Enumerate project_repositories instead of projects for verification/replication of Git repos

Chloe Fonsrequested to merge
546175-geo-enumerate-project_repositories-instead-of-projects-for-git-repos into master

What does this MR do and why?

This change updates Geo to enumerate the project_repositories table instead of the projects table. Since a ProjectRepository contains an actual Git repository (unlike a Project which may not have one), this approach simplifies the logic for determining when a Git repo needs to be fetched for verification and replication.

Under the geo_project_repository_replication_v2 feature flag, this MR:

  • Ensures the necessary tables and triggers are in place for events and states on the ProjectRepository table
  • Associates the ProjectRepositoryReplicator to ProjectRepository instead of Project
  • Ensures the registry can handle ProjectRepository records
  • Ensures this is backwards compatible

NB: There will be a follow-up MR with specs update to ensure both V1 and V2 are tested where projects are concerned.

References

Related to #546175

How to set up and validate locally

  1. Ensure your GDK has got Geo enabled.
  2. Navigate to the admin area to monitor the project repositories verification and sync states
  • /admin/data_management/projects
  • /admin/geo/sites
  1. Enable the feature flag in the Rails console: Feature.enable(:geo_project_repository_replication_v2)
  2. Check up the project repository records:
  • /admin/data_management/project_repositories: the records should be automatically populated and re-checksummed
  • /admin/geo/sites and /admin/geo/sites/2/replication/project_repositories: shouldn't change
  1. Create a new project without a repository through the Rails console:
user = User.first # should be root
namespace = user.namespace
organization = namespace.organization
project = Project.create!(
  name: 'test-no-repo',
  path: 'test-no-repo',
  namespace: namespace,
  creator: user,
  visibility_level: Gitlab::VisibilityLevel::PRIVATE,
  organization: organization
)
  1. Verify the new project does not have a project_repository with project.project_repository. It will return nil.
  2. Verify that this project is not verified or replicated:
  • It does not appear in /admin/data_management/project_repositories
  • It does not appear in /admin/geo/sites/2/replication/project_repositories
  1. Create projects manually through the UI, or import a new project and verify that the ProjectRepository record is created, and that it is successfully replicated.
  2. Now switch the flag off again with `Feature.disable(:geo_project_repository_replication_v2)
  3. Navigate to the admin area to monitor the project repositories verification and sync states
  • /admin/data_management/projects
  • /admin/geo/sites/2/replication/project_repositories
  • the newly added projects should show up and be successfully verified once the workers pick them up.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chloe Fons

Merge request reports