Read Secrets Manager instance enrollment uncached
What does this MR do and why?
On a self-managed instance with more than one Puma worker, the GitLab Secrets Manager toggle in Admin > Settings > General shows the old state for up to a minute after it was flipped. For the same minute, the group and project Secrets pages and CI keep the previous gate on the other workers.
The cause: SecretsManagement::InstanceEnrollment read secrets_manager_instance_enrolled through Gitlab::CurrentSettings. ApplicationSetting is cached per Puma process (Gitlab::ProcessMemoryCache, application_settings_cache_seconds, default 60, see CacheableAttributes#cache!). InstanceEnrollmentService expired only the cache of the worker that served the mutation.
This MR:
- Reads the three enrollment columns (
secrets_manager_instance_enrolled,secrets_manager_instance_beta_enrolled,secrets_manager_instance_add_on_requested_at) from the row itself, as a read-only record memoized once per request inSafeRequestStore, likeAi::Setting.read_only_setting. When no settings row is persisted yet, returns a read-only copy ofGitlab::CurrentSettings, so no caller can write through the shared record. AGitlab::FakeApplicationSettings(rake with pending migrations) is returned as is, since it is not ActiveRecord and cannot persist. - Drops that memo whenever
InstanceEnrollmentServicewrites the settings, and inInstanceEnrollment.stamp_add_on_intent!, so a later read in the same request sees the write. - Resets the process-cached settings record before
InstanceEnrollmentService#update_settingswrites to it. The guards now read the row, so a stale worker whose cached copy already holds the requested value would otherwise let dirty tracking drop the column: the toggle would report success and audit the change while the row stayed as it was.
This also closes a second stale path: after a toggle, Entitlement::Resolver.clear_cache(nil) drops the shared entitlement entry, and a worker recomputing it from stale process settings could write the old answer back to Rails.cache for JOB_PICKUP_CACHE_TTL (1 minute). The recompute now reads the row.
Backend only. The Vue card needs no change once every worker answers with the current state.
Resolves https://gitlab.com/gitlab-org/gitlab/-/work_items/630924
Database
One new read, at most once per request or Sidekiq job, the first time the enrollment is checked. Later checks in the same request are memoized. It runs wherever InstanceEnrollment.enrolled? is called: SecretsManagement::Availability (group and project Secrets pages, CI secret resolution), the Secrets Manager GraphQL resolvers and mutations, and the entitlement resolver.
SELECT "application_settings"."secrets_manager_instance_enrolled",
"application_settings"."secrets_manager_instance_beta_enrolled",
"application_settings"."secrets_manager_instance_add_on_requested_at"
FROM "application_settings"
ORDER BY "application_settings"."id" DESC
LIMIT 1application_settings has a single row, so the read touches one row whatever plan Postgres picks. It is the same ORDER BY id DESC LIMIT 1 that ApplicationSetting.current_without_cache already runs on every cache miss, narrowed to three columns. The update_all calls Danger flagged are in specs only.
With DB load balancing, the read can go to a replica unless the session is stuck to the primary. Replica lag is short but not zero; it is far below the previous 60-second window.
Alternatives considered
Gitlab::ProcessMemoryCache::Helper(used byCi::InstanceVariable): keeps the per-process copy and invalidates it across processes through a Redis timestamp. It saves the database read but costs a Redis read per check, and adds a second cache layer to invalidate. The per-request read is simpler, and the query returns one tiny row.- Fixing
ApplicationSettingcaching for every toggle: out of scope. Every setting read throughGitlab::CurrentSettingshas the same lag of up to a minute, and that fix belongs to the owning team.
How to set up and validate locally
The bug needs more than one Puma worker. GDK runs two by default (workers 2 in the generated config/puma.rb, unless PUMA_SINGLE_MODE is set). The admin setting cache lasts application_settings_cache_seconds (60 in config/gitlab.yml).
Setup:
- Run GDK as self-managed. The instance card is hidden while simulating SaaS, so set
GITLAB_SIMULATE_SAAS: '0'underenv:ingdk.yml, thengdk reconfigure && gdk restart. - Enable the instance enrollment flag and, to get the plain on/off toggle instead of the trial and GitLab Credits buttons (which need CustomersDot), disable the paid experience:
The toggle reads and writes the same
bin/rails runner "Feature.enable(:secrets_manager_instance_enrollment); Feature.disable(:secrets_manager_paid_experience)"secrets_manager_instance_enrolledcolumn throughInstanceEnrollment.enrolled?andInstanceEnrollmentService, so it exercises the change.
Steps, on master and then on this branch (gdk restart rails-web after each checkout):
- As an administrator, open Admin > Settings > General > GitLab Secrets Manager and reload a few times so both workers cache the current state.
- Flip the toggle and reload repeatedly within 60 seconds.
Validated locally (GDK, 2 Puma workers)
| Branch | After switching the toggle off, then reloading within 60 seconds |
|---|---|
master |
Reproduced: a later reload showed the toggle on again |
| This branch | Every reload showed it off (7 of 7) |
For a deterministic check that does not depend on which worker a reload reaches, query the state several times with an admin token (read_api) right after flipping the toggle:
for i in $(seq 10); do curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"query":"{ instanceSecretsManagerEnrollment { enrolled } }"}' http://gdk.test:3000/api/graphql; echo; doneOn master the answers are mixed; with this MR they all match.
Specs:
ee/spec/models/secrets_management/instance_enrollment_spec.rb: with the request store active and the process-cached settings stale,enrolled?,beta_enrolled?andadd_on_requested?all read the row. Each fails if its method goes back toGitlab::CurrentSettings. Also covers: only the three columns are selected and the record is read-only, one read per request, a re-read afterexpire_settings, the no-row fallback (a read-only copy, and the rake-time fake returned as is), and the add-on stamp being visible in the same request.ee/spec/services/secrets_management/instance_enrollment_service_spec.rb: enroll and unenroll are visible to a later read in the same request (both fail withoutexpire_settings). With a stale cached copy that already holds the target value, unenroll and enroll still persist to the row (both fail without the reset). A revert that changed nothing does not write the settings record.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.