Implement Geo support of Helm Metadata Cache behind a feature flag

🛠️ What does this MR do and why?

Helm Metadata Cache Geo support is required for GA of the Helm Chart Registry, ensuring that metadata caches are properly replicated and verified across Geo-distributed GitLab instances.

This MR implements Geo support of Helm Metadata Caches behind a feature flag by adding the necessary models, replicators, and registries.

This MR is part of the whole Helm metadata cache Geo support. The work is split into the following MRs:

  1. Database migration - Create necessary tables and schema (!217447 (merged)) Merged
  2. Model changes - Add Geo models, Registry, Replicator, and feature flags 👈🏻 This MR
  3. Documentation updates - Update API docs and monitoring metrics 👈🏻 This MR
  4. GraphQL API - Add GraphQL mutations/queries for Helm Metadata Cache management
  5. Batch destroy handling - Implement batch destruction logic for Helm Metadata Caches

Note: Step 2 and 3 need to be merged together because of the doc changes.

🗄️ Changes in this MR

This MR implements the following components:

Models and Associations

  • Add Geo::ReplicableModel and Geo::VerifiableModel to Packages::Helm::MetadataCache
  • Create Geo::PackagesHelmMetadataCacheState model for verification state tracking
  • Implement replicables_for_current_secondary scope for selective sync support

Replicator

  • Create Geo::PackagesHelmMetadataCacheReplicator with blob replication strategy
  • Implement carrierwave_uploader method for file handling
  • Add human-readable titles for UI display

Registry

  • Create Geo::PackagesHelmMetadataCacheRegistry model
  • Implement registry factory with traits for different states (synced, failed, verification_succeeded, etc.)

Feature Flag

  • Generate feature flag geo_packages_helm_metadata_cache_replication

Tests

  • Add shared examples for verifiable model
  • Create replicator specs
  • Create registry specs and factory

Database

Packages::Helm::MetadataCache.available_verifiables

Raw query
SELECT
    "packages_helm_metadata_caches".*
FROM
    "packages_helm_metadata_caches"
    INNER JOIN "packages_helm_metadata_cache_states" ON "packages_helm_metadata_cache_states"."packages_helm_metadata_cache_id" = "packages_helm_metadata_caches"."id"

Query plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/48063/commands/144738

Packages::Helm::MetadataCache.pluck_verifiable_ids_in_range(1...10000)

Raw query
SELECT
    "packages_helm_metadata_caches"."id"
FROM
    "packages_helm_metadata_caches"
WHERE
    "packages_helm_metadata_caches"."id" >= 1
    AND "packages_helm_metadata_caches"."id" < 10000

Query plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/48063/commands/144739

Packages::Helm::MetadataCache.replicables_for_current_secondary(1...10000

Raw query
SELECT
    "packages_helm_metadata_caches".*
FROM
    "packages_helm_metadata_caches"
WHERE
    "packages_helm_metadata_caches"."id" >= 1
    AND "packages_helm_metadata_caches"."id" < 10000

Query plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/48063/commands/144740

References

Screenshots or screen recordings

N/A

How to set up and validate locally

  1. Switch to this branch

  2. Run tests:

    bundle exec rspec ee/spec/models/geo/packages_helm_metadata_cache_state_spec.rb
    bundle exec rspec ee/spec/replicators/geo/packages_helm_metadata_cache_replicator_spec.rb
    bundle exec rspec ee/spec/models/geo/packages_helm_metadata_cache_registry_spec.rb
  3. Verify Geo sync (optional - requires Geo setup):

    • Enable the feature flag:

      Feature.enable(:geo_packages_helm_metadata_cache_replication)
    • On Primary Site:

      # Find project and check existing Helm Metadata Caches
      project = Project.find(YOUR_PROJECT_ID)
      project.helm_metadata_caches
    • On Secondary Site:

      # Check if Helm Metadata Cache synced from primary
      project = Project.find(YOUR_PROJECT_ID)
      project.helm_metadata_caches
    • Back on Primary Site:

      # Destroy the Helm Metadata Cache
      project.helm_metadata_caches.destroy_all
    • On Secondary Site:

      # Verify it's no longer present
      project.reload
      project.helm_metadata_caches
    • Back on Primary Site:

      # Create a new Helm Metadata Cache if it's not existed
      ::Packages::Helm::CreateMetadataCacheService
        .new(project, 'stable')
        .execute
      
      # Verify it was created
      project.helm_metadata_caches
    • On Secondary Site:

      # Verify it's present again
      project.helm_metadata_caches

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #582314 (closed)

Edited by Sylvia Shen

Merge request reports

Loading