Geo: Migrate feature flagged settings to database
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem
Geo uses long-lived ops feature flags as permanent admin controls:
- A
geo_<data_type>_replicationand ageo_<data_type>_force_primary_checksummingflag for each data type geo_object_storage_verification
Feature flags work poorly as permanent settings. Their state is visible only in the Rails console, overrides survive upgrades without anyone noticing (see #627766), and admins can't change them from the Admin area or the settings API.
Proposal
This issue is step 1 of &6388: GitLab-wide settings. Site-specific settings are step 2.
The flags become instance-level application settings, stored in one geo_settings JSONB column on application_settings. Each setting applies to every Geo site, the same way the flags work today: feature_gates lives in the primary database, which replicates to the secondaries. Each process caches the settings for up to application_settings_cache_seconds (60 seconds by default), so a change takes effect on a site up to that long after it reaches the site's database. On secondaries, database replication lag comes on top of that, so the cache lifetime alone doesn't guarantee when a change applies.
Scope
Moves to settings:
geo_<data_type>_replicationandgeo_<data_type>_force_primary_checksummingfor each generally available data typegeo_object_storage_verification
Out of scope:
- Site-specific settings, which are step 2.
- Debian project component files and supply chain attestations, which keep their flags until GA (step 6).
- Kill switches and rollout flags, which follow the normal flag lifecycle.
Implementation plan
-
Settings: the
geo_settingscolumn holds a replication key and a primary checksumming key for each data type, plus an object storage verification key. A missing key reads astrue. The column's JSON schema matches the data type keys withpatternProperties, so a new data type needs no schema change, and the validator sets asize_limit. -
Data migration: a regular migration copies each flag's state into the setting, following Migrate an
opsfeature flag to an application setting. A post-deployment migration doesn't work here: between the code deploy and the migration, every key would read astrue, and disabled data types would start replicating again.- The migration hardcodes the flag names and their YAML defaults, instead of deriving them from
Gitlab::Geo::REPLICATOR_CLASSES, which changes over time. - An unset flag has no
featuresrow and gets its YAML default. A flag with afeaturesrow is evaluated from its gates, the same wayFeature.current_feature_valuedoes at runtime (lib/feature.rb), and never falls back to the YAML default.Feature.disabledeletes a flag's gates, so a disabled flag is afeaturesrow with nofeature_gatesrow, and it migrates asfalse. - The replication flags and
geo_object_storage_verificationare checked without an actor, so only a booleantruegate enables them. - The checksumming flags are checked with the
:instanceactor, so either a booleantruegate or anactorsgate for this instance enables them. The instance's actor ID isFlipperGitlabInstance:<host>, built fromGitlab.config.gitlab.host(Feature::FlipperGitlabInstance), and a gate for a different host doesn't count. An opt-out for this instance (FlipperGitlabInstance:<host>:opt_out) overrides both and migrates asfalse. FeatureFlagMigratorHelpersdon't follow these rules, so the migration can't use them unchanged.- The migration doesn't copy flags for Debian project component files or supply chain attestations, which keep their flags until GA (step 6).
- The migration leaves the
featuresandfeature_gatesrows in place, so secondaries still on the previous version behave the same during the upgrade. - Migration specs cover unset flags, explicitly enabled and disabled flags,
actorsgates for matching and non-matching hosts, instance opt-outs, and a fresh install with noapplication_settingsrow. For each case, the spec checks the migrated value againstFeature.enabled?.
- The migration hardcodes the flag names and their YAML defaults, instead of deriving them from
-
Readers: these call sites switch from the flags to the settings:
Gitlab::Geo::Replicator.replication_enabled?inee/lib/gitlab/geo/replicator.rbGeo::VerifiableReplicator#force_primary_checksumming_enabled?inee/app/models/concerns/geo/verifiable_replicator.rbgeo_object_storage_verificationinee/app/models/concerns/geo/replicable_model.rbandee/lib/gitlab/geo/replication/blob_downloader.rb
primary_checksumming_enabled?keeps its current logic: the primary checksums a data type when replication is on, or when forced checksumming is on. The checksumming setting therefore only matters while replication is off, and specs cover all four combinations.Specs that stub these flags, including the replicator shared examples, switch to a shared helper that stubs the settings.
-
API:
GETandPUT /application/settingsexpose the settings as onegeo_settingshash, likedependency_management_settings.doc/api/settings.mddocuments the hash once, so new data types need no API or doc change.GETreturns the effective value of every supported data type, including keys that were never set, so clients learn the supported keys from it.PUTmerges a partial hash into the stored value, so a request can change one data type without resending the others. The update locks the settings row (with_lock) while it reads, merges, and saves, so two concurrent requests that change different data types don't overwrite each other. A spec covers that case.- The API rejects
null, non-boolean values, and unknown data type names, because the JSON schema'spatternPropertiesonly checks the shape of a name. The API checks the keys in each request against the list of data types, instead of validating the whole stored column, so removing a data type later doesn't break saving settings. - An empty hash changes nothing. The default is always
true, so sendingtruerestores a key's default. - The settings don't include keys for Debian project component files or supply chain attestations until those data types reach GA (step 6).
GETdoesn't return them, andPUTrejects them as unknown, so an admin can't save a setting that the readers would ignore.
-
UI: the Admin > Geo > Settings page gets two toggles for each data type, "Replicate" and "Checksum even when replication is disabled", plus a toggle for object storage verification. The page already saves through the settings API, and
doc/administration/geo_sites.mddocuments the new toggles. The page gets the list of data types and their rollout state from the backend (Gitlab::Geo::REPLICATOR_CLASSES, minus the two data types that keep their flags), not from a hardcoded list. The layout needs UX input because there are 44 data types. UX also needs to decide whether to warn admins that secondaries can lose functionality when they disable a data type. -
New data types: each data type added after this issue ships with its setting keys (default
true) and a short-lived rollout flag (default off). The rollout flag gates both replication and forced checksumming: while it's off, Geo neither replicates nor checksums the data type, and the UI hides its toggles. The flag is removed at GA. The replicator generator (ee/lib/generators/geo/replicator_generator.rb) creates the rollout flag instead of the two ops flags, and the SSF issue templates (.gitlab/issue_templates/Geo Replicate a new *.md) anddoc/development/geo/framework.mddescribe the pattern.Debian project component files and supply chain attestations are still rolling out, and each has two independent flags. They keep both flags until GA, so every existing combination keeps working, including replication off with forced checksumming on. At GA, a follow-up moves each one to the settings with the same migration as the other data types. Until then, the readers read the flags for those two data types.
-
Cleanup:
- The YAML files for the moved flags are removed.
- The pages
datatypes.md,object_storage.md,container_registry.md,troubleshooting/synchronization_verification.md, anddoc/development/geo/upload_partition_replication_testing.mddescribe the settings instead ofFeature.enableandFeature.disable. - The 28 field descriptions in
ee/app/graphql/types/geo/geo_node_type.rbthat mention the replication flags, and the comment inee/app/replicators/geo/container_repository_replicator.rb, refer to the settings instead. The GraphQL reference docs are regenerated. - An upgrade note says that the migration keeps the current flag state, that
Feature.disablecommands for the moved flags no longer have any effect, and that a setting changed before every site is upgraded reaches only the upgraded sites. - A post-deployment migration deletes the
featuresandfeature_gatesrows for the moved flags. It ships in the first release after the next required upgrade stop that follows the copy migration. Every supported upgrade path then runs the copy migration, and puts every site on code that reads the settings, before the rows are deleted.
Suggested MRs
- Backend: column, migration, readers, rollout pattern, API, flag removal, and docs
- Frontend: toggles on Admin > Geo > Settings
Related
- #627766: after this issue ships, the moved flags show as settings on Admin > Geo > Settings, so its check only needs to cover the remaining Geo flags.