Geo: Migrate feature flagged settings to database

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem

Geo uses long-lived ops feature flags as permanent admin controls:

  • A geo_<data_type>_replication and a geo_<data_type>_force_primary_checksumming flag for each data type
  • geo_object_storage_verification

Feature flags work poorly as permanent settings. Their state is visible only in the Rails console, overrides survive upgrades without anyone noticing (see #627766), and admins can't change them from the Admin area or the settings API.

Proposal

This issue is step 1 of &6388: GitLab-wide settings. Site-specific settings are step 2.

The flags become instance-level application settings, stored in one geo_settings JSONB column on application_settings. Each setting applies to every Geo site, the same way the flags work today: feature_gates lives in the primary database, which replicates to the secondaries. Each process caches the settings for up to application_settings_cache_seconds (60 seconds by default), so a change takes effect on a site up to that long after it reaches the site's database. On secondaries, database replication lag comes on top of that, so the cache lifetime alone doesn't guarantee when a change applies.

Scope

Moves to settings:

  • geo_<data_type>_replication and geo_<data_type>_force_primary_checksumming for each generally available data type
  • geo_object_storage_verification

Out of scope:

  • Site-specific settings, which are step 2.
  • Debian project component files and supply chain attestations, which keep their flags until GA (step 6).
  • Kill switches and rollout flags, which follow the normal flag lifecycle.

Implementation plan

  1. Settings: the geo_settings column holds a replication key and a primary checksumming key for each data type, plus an object storage verification key. A missing key reads as true. The column's JSON schema matches the data type keys with patternProperties, so a new data type needs no schema change, and the validator sets a size_limit.

  2. Data migration: a regular migration copies each flag's state into the setting, following Migrate an ops feature flag to an application setting. A post-deployment migration doesn't work here: between the code deploy and the migration, every key would read as true, and disabled data types would start replicating again.

    • The migration hardcodes the flag names and their YAML defaults, instead of deriving them from Gitlab::Geo::REPLICATOR_CLASSES, which changes over time.
    • An unset flag has no features row and gets its YAML default. A flag with a features row is evaluated from its gates, the same way Feature.current_feature_value does at runtime (lib/feature.rb), and never falls back to the YAML default. Feature.disable deletes a flag's gates, so a disabled flag is a features row with no feature_gates row, and it migrates as false.
    • The replication flags and geo_object_storage_verification are checked without an actor, so only a boolean true gate enables them.
    • The checksumming flags are checked with the :instance actor, so either a boolean true gate or an actors gate for this instance enables them. The instance's actor ID is FlipperGitlabInstance:<host>, built from Gitlab.config.gitlab.host (Feature::FlipperGitlabInstance), and a gate for a different host doesn't count. An opt-out for this instance (FlipperGitlabInstance:<host>:opt_out) overrides both and migrates as false.
    • FeatureFlagMigratorHelpers don't follow these rules, so the migration can't use them unchanged.
    • The migration doesn't copy flags for Debian project component files or supply chain attestations, which keep their flags until GA (step 6).
    • The migration leaves the features and feature_gates rows in place, so secondaries still on the previous version behave the same during the upgrade.
    • Migration specs cover unset flags, explicitly enabled and disabled flags, actors gates for matching and non-matching hosts, instance opt-outs, and a fresh install with no application_settings row. For each case, the spec checks the migrated value against Feature.enabled?.
  3. Readers: these call sites switch from the flags to the settings:

    • Gitlab::Geo::Replicator.replication_enabled? in ee/lib/gitlab/geo/replicator.rb
    • Geo::VerifiableReplicator#force_primary_checksumming_enabled? in ee/app/models/concerns/geo/verifiable_replicator.rb
    • geo_object_storage_verification in ee/app/models/concerns/geo/replicable_model.rb and ee/lib/gitlab/geo/replication/blob_downloader.rb

    primary_checksumming_enabled? keeps its current logic: the primary checksums a data type when replication is on, or when forced checksumming is on. The checksumming setting therefore only matters while replication is off, and specs cover all four combinations.

    Specs that stub these flags, including the replicator shared examples, switch to a shared helper that stubs the settings.

  4. API: GET and PUT /application/settings expose the settings as one geo_settings hash, like dependency_management_settings. doc/api/settings.md documents the hash once, so new data types need no API or doc change.

    • GET returns the effective value of every supported data type, including keys that were never set, so clients learn the supported keys from it.
    • PUT merges a partial hash into the stored value, so a request can change one data type without resending the others. The update locks the settings row (with_lock) while it reads, merges, and saves, so two concurrent requests that change different data types don't overwrite each other. A spec covers that case.
    • The API rejects null, non-boolean values, and unknown data type names, because the JSON schema's patternProperties only checks the shape of a name. The API checks the keys in each request against the list of data types, instead of validating the whole stored column, so removing a data type later doesn't break saving settings.
    • An empty hash changes nothing. The default is always true, so sending true restores a key's default.
    • The settings don't include keys for Debian project component files or supply chain attestations until those data types reach GA (step 6). GET doesn't return them, and PUT rejects them as unknown, so an admin can't save a setting that the readers would ignore.
  5. UI: the Admin > Geo > Settings page gets two toggles for each data type, "Replicate" and "Checksum even when replication is disabled", plus a toggle for object storage verification. The page already saves through the settings API, and doc/administration/geo_sites.md documents the new toggles. The page gets the list of data types and their rollout state from the backend (Gitlab::Geo::REPLICATOR_CLASSES, minus the two data types that keep their flags), not from a hardcoded list. The layout needs UX input because there are 44 data types. UX also needs to decide whether to warn admins that secondaries can lose functionality when they disable a data type.

  6. New data types: each data type added after this issue ships with its setting keys (default true) and a short-lived rollout flag (default off). The rollout flag gates both replication and forced checksumming: while it's off, Geo neither replicates nor checksums the data type, and the UI hides its toggles. The flag is removed at GA. The replicator generator (ee/lib/generators/geo/replicator_generator.rb) creates the rollout flag instead of the two ops flags, and the SSF issue templates (.gitlab/issue_templates/Geo Replicate a new *.md) and doc/development/geo/framework.md describe the pattern.

    Debian project component files and supply chain attestations are still rolling out, and each has two independent flags. They keep both flags until GA, so every existing combination keeps working, including replication off with forced checksumming on. At GA, a follow-up moves each one to the settings with the same migration as the other data types. Until then, the readers read the flags for those two data types.

  7. Cleanup:

    • The YAML files for the moved flags are removed.
    • The pages datatypes.md, object_storage.md, container_registry.md, troubleshooting/synchronization_verification.md, and doc/development/geo/upload_partition_replication_testing.md describe the settings instead of Feature.enable and Feature.disable.
    • The 28 field descriptions in ee/app/graphql/types/geo/geo_node_type.rb that mention the replication flags, and the comment in ee/app/replicators/geo/container_repository_replicator.rb, refer to the settings instead. The GraphQL reference docs are regenerated.
    • An upgrade note says that the migration keeps the current flag state, that Feature.disable commands for the moved flags no longer have any effect, and that a setting changed before every site is upgraded reaches only the upgraded sites.
    • A post-deployment migration deletes the features and feature_gates rows for the moved flags. It ships in the first release after the next required upgrade stop that follows the copy migration. Every supported upgrade path then runs the copy migration, and puts every site on code that reads the settings, before the rows are deleted.

Suggested MRs

  1. Backend: column, migration, readers, rollout pattern, API, flag removal, and docs
  2. Frontend: toggles on Admin > Geo > Settings
  • #627766: after this issue ships, the moved flags show as settings on Admin > Geo > Settings, so its check only needs to cover the remaining Geo flags.
Edited by Chloe Fons