Replay read-only Secrets Manager entitlement during CDot outages

What does this MR do and why?

Related to #631312. This MR is reviewed in parallel with the flag removal series. It should ideally merge before or with !260110 (merged), but it does not block the other MRs in that series.

Problem

"Replay" means SecretsManagement::Entitlement::Resolver reuses a saved backup copy of the last known good CustomersDot answer, instead of a fresh one, while CustomersDot is down.

  • resolve_from_last_known_good only replays this backup after a Gitlab::SubscriptionPortal::SecretsManagerResponseError::UnavailableError. It is a separate, 24 hour, Redis-backed fallback (LastKnownGoodStore), not the normal Rails.cache entitlement cache.
  • Before replaying, it checked permits_direct_read?. That check was added in commit 2ecbdabfe547, while the beta program was still running and trial_eligible answers still passed it.
  • Since the beta ended (beta_program_ended is always true, see !258441 (merged)), trial_eligible answers fail that check. Blocked answers outside the grace window also fail it.
  • The result: during an outage, those namespaces fall through to fail closed (:ineligible) and lose all access, including the read-only access (secret names and metadata in the UI and GraphQL) that the live answer still gives them. The outage path ends up stricter than the live path.

Where this applies:

Environment Affected?
GitLab.com Yes, today (paid experience is on)
Self-Managed, online, cloud license Yes, once !260110 (merged) removes secrets_manager_paid_experience. The cache key there is instance-wide, so one outage hits every group.
Self-Managed, air-gapped No. It never calls CustomersDot.

Solution

  • Main change: the replay gate now checks permits_read? instead of permits_direct_read?. This is a one line change in ee/lib/secrets_management/entitlement/resolver.rb, plus a 2-line comment explaining why.
  • Unchanged: only UnavailableError triggers the fallback, a fresh refresh never replays, the 24 hour window stays the same, the kill switch secrets_manager_entitlement_lkg_fallback is untouched, and grace re-derivation on replay still happens.
  • Spec changes in ee/spec/lib/secrets_management/entitlement/resolver_spec.rb:
    • The trial_eligible case ("does not replay the slot since the beta program has ended") now expects a read-only replay instead: state trial_eligible, permits_read? true, permits_direct_read? false, permits_writes? false. The same case was added for a non-beta namespace, using a shared example for both.
    • A stored blocked deny (credits_exhausted) on a 5xx response and on a transport failure is now replayed as blocked read-only, keeping the same blocked_reason, with no direct reads and no writes. These used to expect :ineligible.
    • The grace window lapsing during an outage (on SaaS and Self-Managed) now expects blocked (not grace), with no direct reads but with read access, instead of :ineligible. CI reads still stop when grace lapses, which is what these specs protect.

Why this is safe

A replay never grants more access than the live answer, because every caller that needs more than read-only still checks direct reads or writes on the replayed entitlement:

  • The CI runner payload (BuildRunnerPresenter) checks permits_direct_read?.
  • Job scheduling (EE::Ci::RegisterJobService) checks permits_direct_read?.
  • The access token API is gated by create_secrets_manager_api_jwt, which the policy blocks via secrets_manager_entitlement_denies_direct_read.
  • Writes check permits_writes? (in EntitlementGate and the policies).

Tradeoff

A replayed blocked answer keeps showing its stored blocked reason (for example credits_exhausted) for up to 24 hours of outage, even if the customer buys credits during that outage. Before this change, the namespace showed as ineligible instead. Access never goes above what the stored answer allows.

Impact

No database changes.

Changelog: commit trailer Changelog: fixed and EE: true.

References

Screenshots or screen recordings

Not applicable. No UI change.

How to set up and validate locally

  1. Run bundle exec rspec ee/spec/lib/secrets_management/entitlement/resolver_spec.rb ee/spec/lib/secrets_management/entitlement_spec.rb.

Locally, resolver_spec and entitlement_spec pass with 341 examples and 0 failures. Related SM specs (last_known_good_store, effective_capabilities_service, start_trial, user_permissions, refresh_entitlement) pass with 204 examples and 0 failures. Rubocop is clean on the changed files.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Erick Bajao

Merge request reports

Loading
Loading