## Update 2026-07-07: rescoped after looking at the current code Most of this proposal turned out to be already done or not worth doing. Here is where each phase actually stands, and what shipped. ### Phase status - **Phase 5 (Rails generator): already done.** `rails g geo:blob_replicator --upload-partition` (and `geo:repository_replicator`) already generate the full set of files, patch the shared framework files, and regenerate docs/dictionaries/GraphQL. `--only-post-generate` re-applies the patches after a rebase. - **Phase 3 (base upload replicator): already done, via composition.** The shared behaviour lives in `Geo::Concerns::UploadReplicatorBehavior` and `Geo::PartitionUploadRegistry`, which every upload replicator includes. A base class would move us away from composition, so we are keeping the concerns. - **Phase 4 (shared registry table): not doing this.** We recently did the opposite: the single `file_registry`-backed registry was split into per-partition tables across 18.10 to 19.2, and those tables are deployed. Re-consolidating would mean migrating and backfilling live tables on secondaries, giving up the per-partition foreign keys (each state table has a hard FK to its leaf `*_uploads` partition), and it collides with the in-flight `uploads_archived` cleanup. The boilerplate here is voluminous but uniform and generator-maintained, so the cost of keeping it is low. - **Phase 1 (auto-discovery): not doing the disk-traversal version.** Removing `REPLICATOR_CLASSES` entirely was explored in #227693 and hits a real problem: `Gitlab::Geo::Replicator.subclasses` returns a partial list under `eager_load = false` (dev/console), plus reviewer pushback on scanning the filesystem. Instead we keep the explicit list as the single source of truth and derive the other registration points from it. - **Phase 2 (dynamic GraphQL): partially done.** `RegistryClassEnum` already loops the list, and we derived the `RegistrableType` union too. We are deliberately not auto-generating the `GeoNodeType` registry fields: their descriptions and `experiment` milestones are bespoke per field (genuine API content, not boilerplate), so generating them would either change the public schema or just move the strings around. ### What shipped Keeping `REPLICATOR_CLASSES` as the single source of truth and deriving the parallel registration points from it: - !243966 - derive the consistency worker's `REGISTRY_CLASSES` from `REPLICATOR_CLASSES`, and add a conformance spec that fails loudly if a replicator is missing a registry or factory. - !243992 - derive the `RegistrableType` GraphQL union from `REPLICATOR_CLASSES`, with a `graphql_registerable?` opt-out for replicators without a GraphQL type. Stacked on !243966. - !243982 - move the identical `carrierwave_uploader` method into `UploadReplicatorBehavior` instead of repeating it in 24 replicators. Each of these also drops a step the generator had to hand-patch, so adding a new upload replicator now touches fewer shared files. --- _Original proposal below._ > **📋 Phase 1: Foundation & First Replicator (POC)** | Risk: Low | [View Epic &20933](https://gitlab.com/groups/gitlab-org/-/work_items/20933) --- ## Summary This issue proposes improvements to the Geo Self-Service Framework (SSF) to reduce boilerplate code when adding new blob replicators, specifically for the 22 upload partition tables that need individual Geo replication support. ## Background Currently, adding a new blob type to Geo replication requires creating multiple files with nearly identical boilerplate code. For each new replicable, developers must create: 1. **Replicator class** (`ee/app/replicators/geo/*_replicator.rb`) 2. **Registry model** (`ee/app/models/geo/*_registry.rb`) 3. **State model** (`ee/app/models/geo/*_state.rb`) 4. **Registry finder** (`ee/app/finders/geo/*_registry_finder.rb`) 5. **GraphQL resolver** (`ee/app/graphql/resolvers/geo/*_registries_resolver.rb`) 6. **GraphQL type** (`ee/app/graphql/types/geo/*_registry_type.rb`) 7. **Database migrations** (registry table in geo DB, state table in main DB) 8. **Database dictionary files** 9. **Factory files** for testing 10. **Spec files** for each of the above 11. **Manual registration** in `REPLICATOR_CLASSES` array in `ee/lib/gitlab/geo.rb` 12. **Manual registration** in `REGISTRY_CLASSES` in `registry_consistency_worker.rb` 13. **Manual updates** to `GeoNodeType` GraphQL type 14. **Manual updates** to `registrable_type.rb` for resync/reverify support With 22 upload partition tables to support, this means creating ~200+ files with mostly identical code. ## Proposal: Dynamic Replicator Generation ### Phase 1: Convention-based Auto-discovery Replace manual registration with convention-based auto-discovery: ```ruby # ee/lib/gitlab/geo.rb def self.replicator_classes @replicator_classes ||= discover_replicator_classes end def self.discover_replicator_classes Dir[Rails.root.join('ee/app/replicators/geo/*_replicator.rb')].map do |file| class_name = "Geo::#{File.basename(file, '.rb').camelize}" class_name.constantize end.select { |klass| klass < Gitlab::Geo::Replicator } end ``` ### Phase 2: Dynamic GraphQL Registration Auto-generate GraphQL types, resolvers, and fields based on registered replicators: ```ruby # ee/app/graphql/types/geo/geo_node_type.rb Gitlab::Geo.replicator_classes.each do |replicator_class| field replicator_class.graphql_field_name, replicator_class.graphql_registry_type.connection_type, null: true, resolver: replicator_class.graphql_resolver_class end ``` ### Phase 3: Base Upload Replicator for Partitioned Tables Create a base class specifically for upload partition replicators: ```ruby # ee/app/replicators/geo/base_upload_partition_replicator.rb module Geo class BaseUploadPartitionReplicator < Gitlab::Geo::Replicator include ::Geo::BlobReplicatorStrategy class << self # Subclasses only need to define: # - model (the upload model class) # - replicable_title / replicable_title_plural def registry_class # Auto-generate or use shared registry with partition key @registry_class ||= generate_registry_class end end def carrierwave_uploader model_record.retrieve_uploader end end end ``` ### Phase 4: Shared Registry with Partition Discrimination Instead of 22 separate registry tables, consider a shared registry approach: ```ruby # Single registry table with upload_type discriminator create_table :geo_upload_partition_registries do |t| t.string :upload_type, null: false # e.g., 'abuse_report', 'achievement', etc. t.bigint :upload_id, null: false # ... standard registry columns t.index [:upload_type, :upload_id], unique: true end ``` This would allow a single `Geo::UploadPartitionRegistry` model with STI or type discrimination. ### Phase 5: Generator Script Create a Rails generator for new upload partition replicators: ```bash bin/rails generate geo:upload_partition_replicator AbuseReport \ --table=abuse_report_uploads \ --sharding_key=organization_id ``` This would generate all necessary files with correct naming and configuration. ## Implementation Checklist ### Auto-discovery and Registration - [ ] Implement convention-based replicator class discovery - [ ] Remove manual `REPLICATOR_CLASSES` array maintenance - [ ] Auto-register registry classes in consistency worker - [ ] Auto-generate GraphQL fields on `GeoNodeType` ### Base Classes and Concerns - [ ] Create `Geo::BaseUploadPartitionReplicator` base class - [ ] Extract common upload replicator logic into shared concern - [ ] Create shared registry concern for upload partitions ### Database Optimization - [ ] Evaluate shared registry table vs individual tables - [ ] Create migration generator for registry/state tables - [ ] Auto-generate database dictionary files ### GraphQL Automation - [ ] Dynamic GraphQL type generation from replicator metadata - [ ] Dynamic resolver generation - [ ] Auto-registration in `registrable_type.rb` ### Testing Infrastructure - [ ] Shared examples that work with minimal configuration - [ ] Factory generator for new replicators - [ ] Automated spec generation ### Documentation and Tooling - [ ] Rails generator for new upload partition replicators - [ ] Update issue template to reflect reduced manual steps - [ ] Document the new streamlined process ## Benefits 1. **Reduced code duplication**: ~90% reduction in boilerplate files 2. **Faster implementation**: Adding a new upload type takes minutes instead of hours 3. **Fewer errors**: Less manual registration means fewer missed steps 4. **Easier maintenance**: Changes to common behavior only need to be made once 5. **Better consistency**: All upload replicators behave identically ## Risks and Mitigations | Risk | Mitigation | |------|------------| | Magic/implicit behavior harder to debug | Good logging, clear documentation | | Performance of auto-discovery at boot | Cache discovered classes, lazy loading | | Breaking existing replicators | Gradual migration, feature flags | ## Related - Parent epic: &20933 - MR: !221773 - Related issue: https://gitlab.com/gitlab-org/gitlab/-/issues/227693 (Avoid maintaining REPLICATOR_CLASSES list)