Read Secrets Manager instance enrollment uncached

What does this MR do and why?

On a self-managed instance with more than one Puma worker, the GitLab Secrets Manager toggle in Admin > Settings > General shows the old state for up to a minute after it was flipped. For the same minute, the group and project Secrets pages and CI keep the previous gate on the other workers.

The cause: SecretsManagement::InstanceEnrollment read secrets_manager_instance_enrolled through Gitlab::CurrentSettings. ApplicationSetting is cached per Puma process (Gitlab::ProcessMemoryCache, application_settings_cache_seconds, default 60, see CacheableAttributes#cache!). InstanceEnrollmentService expired only the cache of the worker that served the mutation.

This MR:

  • Reads the three enrollment columns (secrets_manager_instance_enrolled, secrets_manager_instance_beta_enrolled, secrets_manager_instance_add_on_requested_at) from the row itself, as a read-only record memoized once per request in SafeRequestStore, like Ai::Setting.read_only_setting. When no settings row is persisted yet, returns a read-only copy of Gitlab::CurrentSettings, so no caller can write through the shared record. A Gitlab::FakeApplicationSettings (rake with pending migrations) is returned as is, since it is not ActiveRecord and cannot persist.
  • Drops that memo whenever InstanceEnrollmentService writes the settings, and in InstanceEnrollment.stamp_add_on_intent!, so a later read in the same request sees the write.
  • Resets the process-cached settings record before InstanceEnrollmentService#update_settings writes to it. The guards now read the row, so a stale worker whose cached copy already holds the requested value would otherwise let dirty tracking drop the column: the toggle would report success and audit the change while the row stayed as it was.

This also closes a second stale path: after a toggle, Entitlement::Resolver.clear_cache(nil) drops the shared entitlement entry, and a worker recomputing it from stale process settings could write the old answer back to Rails.cache for JOB_PICKUP_CACHE_TTL (1 minute). The recompute now reads the row.

Backend only. The Vue card needs no change once every worker answers with the current state.

Resolves https://gitlab.com/gitlab-org/gitlab/-/work_items/630924

Database

One new read, at most once per request or Sidekiq job, the first time the enrollment is checked. Later checks in the same request are memoized. It runs wherever InstanceEnrollment.enrolled? is called: SecretsManagement::Availability (group and project Secrets pages, CI secret resolution), the Secrets Manager GraphQL resolvers and mutations, and the entitlement resolver.

SELECT "application_settings"."secrets_manager_instance_enrolled",
       "application_settings"."secrets_manager_instance_beta_enrolled",
       "application_settings"."secrets_manager_instance_add_on_requested_at"
FROM "application_settings"
ORDER BY "application_settings"."id" DESC
LIMIT 1

application_settings has a single row, so the read touches one row whatever plan Postgres picks. It is the same ORDER BY id DESC LIMIT 1 that ApplicationSetting.current_without_cache already runs on every cache miss, narrowed to three columns. The update_all calls Danger flagged are in specs only.

With DB load balancing, the read can go to a replica unless the session is stuck to the primary. Replica lag is short but not zero; it is far below the previous 60-second window.

Alternatives considered

  • Gitlab::ProcessMemoryCache::Helper (used by Ci::InstanceVariable): keeps the per-process copy and invalidates it across processes through a Redis timestamp. It saves the database read but costs a Redis read per check, and adds a second cache layer to invalidate. The per-request read is simpler, and the query returns one tiny row.
  • Fixing ApplicationSetting caching for every toggle: out of scope. Every setting read through Gitlab::CurrentSettings has the same lag of up to a minute, and that fix belongs to the owning team.

How to set up and validate locally

The bug needs more than one Puma worker. GDK runs two by default (workers 2 in the generated config/puma.rb, unless PUMA_SINGLE_MODE is set). The admin setting cache lasts application_settings_cache_seconds (60 in config/gitlab.yml).

Setup:

  1. Run GDK as self-managed. The instance card is hidden while simulating SaaS, so set GITLAB_SIMULATE_SAAS: '0' under env: in gdk.yml, then gdk reconfigure && gdk restart.
  2. Enable the instance enrollment flag and, to get the plain on/off toggle instead of the trial and GitLab Credits buttons (which need CustomersDot), disable the paid experience:
    bin/rails runner "Feature.enable(:secrets_manager_instance_enrollment); Feature.disable(:secrets_manager_paid_experience)"
    The toggle reads and writes the same secrets_manager_instance_enrolled column through InstanceEnrollment.enrolled? and InstanceEnrollmentService, so it exercises the change.

Steps, on master and then on this branch (gdk restart rails-web after each checkout):

  1. As an administrator, open Admin > Settings > General > GitLab Secrets Manager and reload a few times so both workers cache the current state.
  2. Flip the toggle and reload repeatedly within 60 seconds.

Validated locally (GDK, 2 Puma workers)

Branch After switching the toggle off, then reloading within 60 seconds
master Reproduced: a later reload showed the toggle on again
This branch Every reload showed it off (7 of 7)

For a deterministic check that does not depend on which worker a reload reaches, query the state several times with an admin token (read_api) right after flipping the toggle:

for i in $(seq 10); do curl -s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"query":"{ instanceSecretsManagerEnrollment { enrolled } }"}' http://gdk.test:3000/api/graphql; echo; done

On master the answers are mixed; with this MR they all match.

Specs:

  • ee/spec/models/secrets_management/instance_enrollment_spec.rb: with the request store active and the process-cached settings stale, enrolled?, beta_enrolled? and add_on_requested? all read the row. Each fails if its method goes back to Gitlab::CurrentSettings. Also covers: only the three columns are selected and the record is read-only, one read per request, a re-read after expire_settings, the no-row fallback (a read-only copy, and the rake-time fake returned as is), and the add-on stamp being visible in the same request.
  • ee/spec/services/secrets_management/instance_enrollment_service_spec.rb: enroll and unenroll are visible to a later read in the same request (both fail without expire_settings). With a stale cached copy that already holds the target value, unenroll and enroll still persist to the row (both fail without the reset). A revert that changed nothing does not write the settings record.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

🤖 Generated with Claude Code

Edited by Dmytro Biryukov

Merge request reports

Loading
Loading