Geo: Test project repository replication v2

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Integration & unit tests

The idea of this issue is to conduct manual testing of the feature, as well as writing missing tests.

Automated tests

1. Assess current coverage of V1 vs. V2 unit tests for project_repositories and projects
2. Investigate if it's possible to write a fully blown integration test for this feature

Manual test plan: Geo project repository replication V2

Related MRs:

• !194051 (closed) - V2 replication architecture
• !198308 (merged) - Prevent phantom records creation (merged)
• !223027 (closed) - Enumerate project_repositories instead of projects for verification/replication

Feature flags:

• geo_project_repository_replication (existing)
• geo_project_repository_replication_v2 (new)

Test environment requirements

• GitLab Geo setup with primary and secondary nodes
• Admin access to both nodes
• Feature flags available for toggling
• Access to Rails console on both nodes
• Ability to create projects with and without repositories

Test scenarios

Baseline capture (before any flag changes)

Before running any test scenarios, capture the baseline state of the following pages and note the counts and statuses:

• /admin/data_management/projects
• /admin/geo/sites
• /admin/geo/sites/2/replication/project_repositories

This baseline will be used to verify changes and detect regressions throughout testing.

Phase 1: Basic functionality tests

1.1 Projects without repositories should not be replicated

Since !198308 (merged) is already merged, phantom project_repository record creation is prevented regardless of the V2 flag. This scenario verifies the end-to-end behaviour with geo_project_repository_replication_v2 enabled.

# On primary - Rails console
# Enable V2 flag first
Feature.enable(:geo_project_repository_replication_v2)

user = User.first # should be root
namespace = user.namespace
organization = namespace.organization

project = Project.create!(
  name: 'test-no-repo',
  path: 'test-no-repo',
  namespace: namespace,
  creator: user,
  visibility_level: Gitlab::VisibilityLevel::PRIVATE,
  organization: organization
)

project.project_repository # Should return nil

Expected results:

• project.project_repository returns nil
• No Geo replication attempt (no registry created)
• Project does not appear in /admin/geo/sites/2/replication/project_repositories
• No errors in secondary logs

1.2 Projects with repositories should replicate successfully

# On primary - Rails console
user = User.first
project = Projects::CreateService.new(user, {
  name: 'test-with-repo',
  path: 'test-with-repo',
  initialize_with_readme: true
}).execute

After enabling the flag, also verify that /admin/data_management/projects is automatically populated and re-checksummed for existing projects with repositories.

Expected results:

• project_repository record created
• Successful replication to secondary
• Project appears in /admin/geo/sites/2/replication/project_repositories with synced status
• No errors in logs

Phase 2: Feature flag switching tests

Note: Phase 2 tests may be partially blocked by #563394 (Geo: Handle toggling between legacy and v2 of project repo replication). Review that issue's status before running these scenarios.

2.1 V1 → V2 migration

# Start with V2 disabled
Feature.disable(:geo_project_repository_replication_v2)

# Capture baseline state of admin pages (see baseline capture above)

# Create test projects (mix with/without repos)
user = User.first
namespace = user.namespace
organization = namespace.organization

5.times do |i|
  if i.even?
    # With repository
    Projects::CreateService.new(user, {
      name: "migration-test-#{i}",
      path: "migration-test-#{i}",
      initialize_with_readme: true
    }).execute
  else
    # Without repository
    Project.create!(
      name: "migration-test-#{i}",
      path: "migration-test-#{i}",
      namespace: namespace,
      creator: user,
      visibility_level: Gitlab::VisibilityLevel::PRIVATE,
      organization: organization
    )
  end
end

# Wait for replication to complete, then check registry state on secondary

# Enable V2 feature flag
Feature.enable(:geo_project_repository_replication_v2)

# Create more projects and verify behaviour

Expected results:

• Existing registries continue working after flag enable
• New projects with repos use V2 logic and are replicated
• Projects without repos don't create registries in V2 mode
• No duplicate replication
• UI shows consistent counts on /admin/geo/sites

Geo::Event observations (see #563394):

• Creating a project with a repository creates a Geo::Event for the project_repository
• Creating a project without a repository does not create a Geo::Event
• Updating a project (e.g. renaming) creates the expected Geo::Event
• Deleting a project creates the expected Geo::Event and the registry is cleaned up on the secondary

2.2 V2 → V1 rollback

# Start with V2 enabled
Feature.enable(:geo_project_repository_replication_v2)

# Create projects and verify replication (see 1.1 and 1.2 above)

# Disable V2 feature flag
Feature.disable(:geo_project_repository_replication_v2)

# Create new projects and verify fallback

After disabling the flag, check:

• /admin/data_management/projects
• /admin/geo/sites/2/replication/project_repositories

Wait for workers to pick up existing projects and verify they are successfully verified and replicated under V1.

Expected results:

• Existing V2 registries continue working via delegation
• New projects use V1 logic
• No replication interruption
• Newly added projects appear in /admin/data_management/projects and are successfully verified once workers pick them up
• UI remains functional on all admin pages

Geo::Event observations (see #563394):

• After rollback, creating a project with a repository creates a Geo::Event scoped to the project (V1 behaviour)
• No Geo::Event records are orphaned from the V2 period

Phase 3: UI and API tests

3.1 Admin Geo status page

Test steps:

1. Navigate to /admin/geo/sites
2. Check "Project repositories" section in replication status
3. Verify counts and status indicators
4. Test with both feature flag states

Expected results:

• Accurate counts displayed for both V1 and V2
• Status indicators work correctly (synced/failed/pending)
• No GraphQL errors in browser console
• Performance acceptable with large datasets

3.2 GraphQL API compatibility

query {
  geoNode {
    projectRepositoryRegistries {
      nodes {
        id
        projectId          # Should always be present (backward compatibility)
        state
        lastSyncedAt
      }
    }
  }
}

Expected results:

• projectId field always present (backward compatibility)
• No breaking changes for existing API consumers
• Proper error handling for edge cases

Phase 4: Error scenarios and edge cases

4.1 Repository deletion after creation

# Create project with repository
user = User.first
project = Projects::CreateService.new(user, {
  name: 'test-repo-deletion',
  path: 'test-repo-deletion',
  initialize_with_readme: true
}).execute

# Wait for replication
# Delete repository but keep project
project.repository.remove

# Trigger re-verification

Expected results:

• V2: Should handle gracefully, possibly remove registry
• No infinite retry loops
• Proper error messages in logs

4.2 Corrupt registry data

# Create registry with invalid project_repository_id
registry = Geo::ProjectRepositoryRegistry.create!(project_repository_id: 999999, project_id: project.id)

# Trigger replication worker
Geo::ProjectRepositoryReplicator.new(model_record_id: 999999).execute

Expected results:

• Graceful error handling
• No worker crashes
• Proper error logging

Success criteria

Must pass ✅

• All projects with repositories replicate successfully in V2
• Projects without repositories don't cause errors in V2 mode
• Feature flag switching works seamlessly in both directions (pending #563394)
• UI shows accurate status and counts in all scenarios
• No regression in existing Geo functionality
• GraphQL API maintains backward compatibility

Performance ⚡

• No significant performance degradation during normal operations
• Memory usage remains stable during feature flag switches
• Replication throughput maintained or improved

Error handling 🛡️

• Graceful handling of edge cases (missing repos, corrupt data)
• Proper error messages and logging (no cryptic failures)
• No infinite retry loops or worker crashes
• Clear recovery procedures for problematic states