Replay read-only Secrets Manager entitlement during CDot outages
What does this MR do and why?
Related to #631312. This MR is reviewed in parallel with the flag removal series. It should ideally merge before or with !260110 (merged), but it does not block the other MRs in that series.
Problem
"Replay" means SecretsManagement::Entitlement::Resolver reuses a saved backup copy of the last known good CustomersDot answer, instead of a fresh one, while CustomersDot is down.
resolve_from_last_known_goodonly replays this backup after aGitlab::SubscriptionPortal::SecretsManagerResponseError::UnavailableError. It is a separate, 24 hour, Redis-backed fallback (LastKnownGoodStore), not the normalRails.cacheentitlement cache.- Before replaying, it checked
permits_direct_read?. That check was added in commit2ecbdabfe547, while the beta program was still running andtrial_eligibleanswers still passed it. - Since the beta ended (
beta_program_endedis always true, see !258441 (merged)),trial_eligibleanswers fail that check. Blocked answers outside the grace window also fail it. - The result: during an outage, those namespaces fall through to fail closed (
:ineligible) and lose all access, including the read-only access (secret names and metadata in the UI and GraphQL) that the live answer still gives them. The outage path ends up stricter than the live path.
Where this applies:
| Environment | Affected? |
|---|---|
| GitLab.com | Yes, today (paid experience is on) |
| Self-Managed, online, cloud license | Yes, once !260110 (merged) removes secrets_manager_paid_experience. The cache key there is instance-wide, so one outage hits every group. |
| Self-Managed, air-gapped | No. It never calls CustomersDot. |
Solution
- Main change: the replay gate now checks
permits_read?instead ofpermits_direct_read?. This is a one line change inee/lib/secrets_management/entitlement/resolver.rb, plus a 2-line comment explaining why. - Unchanged: only
UnavailableErrortriggers the fallback, a fresh refresh never replays, the 24 hour window stays the same, the kill switchsecrets_manager_entitlement_lkg_fallbackis untouched, and grace re-derivation on replay still happens. - Spec changes in
ee/spec/lib/secrets_management/entitlement/resolver_spec.rb:- The
trial_eligiblecase ("does not replay the slot since the beta program has ended") now expects a read-only replay instead: statetrial_eligible,permits_read?true,permits_direct_read?false,permits_writes?false. The same case was added for a non-beta namespace, using a shared example for both. - A stored blocked deny (
credits_exhausted) on a 5xx response and on a transport failure is now replayed as blocked read-only, keeping the sameblocked_reason, with no direct reads and no writes. These used to expect:ineligible. - The grace window lapsing during an outage (on SaaS and Self-Managed) now expects blocked (not grace), with no direct reads but with read access, instead of
:ineligible. CI reads still stop when grace lapses, which is what these specs protect.
- The
Why this is safe
A replay never grants more access than the live answer, because every caller that needs more than read-only still checks direct reads or writes on the replayed entitlement:
- The CI runner payload (
BuildRunnerPresenter) checkspermits_direct_read?. - Job scheduling (
EE::Ci::RegisterJobService) checkspermits_direct_read?. - The access token API is gated by
create_secrets_manager_api_jwt, which the policy blocks viasecrets_manager_entitlement_denies_direct_read. - Writes check
permits_writes?(inEntitlementGateand the policies).
Tradeoff
A replayed blocked answer keeps showing its stored blocked reason (for example credits_exhausted) for up to 24 hours of outage, even if the customer buys credits during that outage. Before this change, the namespace showed as ineligible instead. Access never goes above what the stored answer allows.
Impact
No database changes.
Changelog: commit trailer Changelog: fixed and EE: true.
References
- Related to #631312
- Commit that added the gate:
2ecbdabfe547 - Beta end MR: !258441 (merged)
- Paid experience MR: !260110 (merged)
Screenshots or screen recordings
Not applicable. No UI change.
How to set up and validate locally
- Run
bundle exec rspec ee/spec/lib/secrets_management/entitlement/resolver_spec.rb ee/spec/lib/secrets_management/entitlement_spec.rb.
Locally, resolver_spec and entitlement_spec pass with 341 examples and 0 failures. Related SM specs (last_known_good_store, effective_capabilities_service, start_trial, user_permissions, refresh_entitlement) pass with 204 examples and 0 failures. Rubocop is clean on the changed files.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.