Add script to generate Geo SSF boilerplate for blob replicators

Douglas Barbosa Alexandrerequested to merge
duo-edit-20260219-230310 into master

Summary

Adds a Ruby script (scripts/geo/generate-blob-replicator) that automates the generation of all boilerplate files required by the Geo Self-Service Framework when adding a new blob replicator.

This addresses the need to create 22+ upload partition replicators with minimal manual effort, as described in &20933 and #589925.

What it generates

New files (17+ per replicator):

  • Replicator class (ee/app/replicators/geo/)
  • Registry model (ee/app/models/geo/)
  • State model (ee/app/models/geo/)
  • Registry finder (ee/app/finders/geo/)
  • GraphQL resolver (ee/app/graphql/resolvers/geo/)
  • GraphQL type (ee/app/graphql/types/geo/)
  • Feature flag definition (ee/config/feature_flags/development/)
  • Geo tracking DB migration (ee/db/geo/migrate/)
  • Main DB migration (db/migrate/)
  • Database dictionary files (ee/db/geo/docs/, db/docs/)
  • Factory files (ee/spec/factories/geo/)
  • Spec files for replicator, registry model, finder, GraphQL resolver, and GraphQL type

With --upload-partition flag, also generates:

  • Dedicated read-only partition model (ee/app/models/geo/) inheriting from Upload
  • Partition model factory (creates via Upload, reads from partition)
  • Partition model spec with selective sync tests
  • Shared example for read-only replicable model blob replicators (generated once, reused)

Patches to existing files (14 files):

  • REPLICATOR_CLASSES in ee/lib/gitlab/geo.rb
  • REGISTRY_CLASSES in ee/app/workers/geo/secondary/registry_consistency_worker.rb
  • GeoNodeType GraphQL fields
  • RegistrableType GEO_REGISTRY_TYPES
  • possible_types.json Registrable array
  • Inflections in config/initializers_before_autoloader/000_inflections.rb
  • Registries shared context
  • Registries spec
  • GeoNodeType spec
  • RegistryClassEnum spec
  • geo_node_status.json and geo_site_status.json schema fixtures
  • doc/api/geo_nodes.md and doc/api/geo_sites.md
  • doc/administration/monitoring/prometheus/gitlab_metrics.md

Usage 

# Standard blob replicator
scripts/geo/generate-blob-replicator \
  --replicable-name=cool_widget \
  --model-class=CoolWidget \
  --table-name=cool_widgets \
  --sharding-key=project_id \
  --milestone=18.10

# Upload partition replicator
scripts/geo/generate-blob-replicator \
  --replicable-name=abuse_report_upload \
  --model-class=AbuseReport \
  --table-name=abuse_report_uploads \
  --sharding-key=organization_id \
  --milestone=18.10 \
  --upload-partition

# Dry run (preview without writing)
scripts/geo/generate-blob-replicator \
  --replicable-name=abuse_report_upload \
  --model-class=AbuseReport \
  --table-name=abuse_report_uploads \
  --sharding-key=organization_id \
  --milestone=18.10 \
  --upload-partition \
  --dry-run

How to set up and validate locally

1. Generate the boilerplate for AbuseReport uploads 

scripts/geo/generate-blob-replicator \
  --replicable-name=abuse_report_upload \
  --model-class=AbuseReport \
  --table-name=abuse_report_uploads \
  --sharding-key=organization_id \
  --milestone=18.10 \
  --upload-partition

2. Run the required manual steps

The script prints a summary of remaining manual steps:

  1. Run bundle exec rake db:migrate:geo and bundle exec rake db:migrate
  2. Commit structure.sql changes
  3. For --upload-partition: review the auto-generated model; for standard mode: add model concerns manually
  4. Run bundle exec rake geo:dev:ssf_metrics
  5. Run bundle exec rake gitlab:graphql:compile_docs

3. Run the generated specs 

# Replicator spec
bundle exec rspec ee/spec/replicators/geo/abuse_report_upload_replicator_spec.rb

# Registry model spec
bundle exec rspec ee/spec/models/geo/abuse_report_upload_registry_spec.rb

# Partition model spec
bundle exec rspec ee/spec/models/geo/abuse_report_upload_spec.rb

# Finder spec
bundle exec rspec ee/spec/finders/geo/abuse_report_upload_registry_finder_spec.rb

# GraphQL resolver spec
bundle exec rspec ee/spec/graphql/resolvers/geo/abuse_report_upload_registries_resolver_spec.rb

# GraphQL type spec
bundle exec rspec ee/spec/graphql/types/geo/abuse_report_upload_registry_type_spec.rb

7. Verify on a Geo setup (optional)

If you have a Geo GDK setup:

  1. On the primary, disable the existing upload replication/verification feature flags:

    The existing Upload replicator must be disabled to avoid conflicts with the new partition replicator during testing:

    # In a Rails console
Feature.disable(:geo_upload_replication)
Feature.disable(:geo_upload_force_primary_checksumming)

  2. On the primary, enable the new partition replication/verification feature flags:

    # In a Rails console
Feature.enable(:geo_abuse_report_upload_replication)
Feature.enable(:geo_abuse_report_force_primary_checksumming)

  3. On the primary, create an AbuseReport with an upload:

    Follow the instructions in https://docs.gitlab.com/user/report_abuse to create an abuse report with a screenshot.

  4. On the secondary, verify the upload was replicated:

    # In a Rails console on the secondary
Geo::AbuseReportUpload.count
Geo::AbuseReportUploadRegistry.count
Geo::AbuseReportUploadRegistry.synced.count

  5. Check the Geo admin page at /admin/geo/sites for the new data type.

  6. Check the Data management page at /admin/data_management for the new data type.

8. Clean up (if testing only)

  1. Clean up generated files:

    git checkout -- .
bundle exec rake db:rollback:main
bundle exec rake db:rollback:ci
bundle exec rake db:rollback:geo

  2. On the primary, disable the new partition replication/verification feature flags:

    # In a Rails console
Feature.disable(:geo_abuse_report_upload_replication)
Feature.disable(:geo_abuse_report_force_primary_checksumming)

  3. On the primary, enable the existing upload replication/verification feature flags:

    # In a Rails console
Feature.enable(:geo_upload_replication)
Feature.enable(:geo_upload_force_primary_checksumming)

Related

Edited by Douglas Barbosa Alexandre

Merge request reports