Stop denying Secrets Manager access when CDot is unavailable

Jayakrishnan Mallisseryrequested to merge
jay-sm-entitlement-lkg-fallback into master

What does this MR do and why?

  • Fixes https://gitlab.com/gitlab-org/gitlab/-/work_items/622853
  • When CustomersDot was unreachable, entitlement resolution failed closed to :ineligible, so an entitled customer's CI jobs lost their secrets and failed with an opaque runner system failure
  • On self-managed the entitlement is install-wide, so a single failure hits every pipeline on the install
  • This MR serves CustomersDot's last answer for up to 24h instead, adds a timeout to every request-path caller, and adds a counter plus one searchable log line so the next outage is diagnosable
  • A denial still needs an explicit CustomersDot answer, so this never grants access CustomersDot did not already grant. Not fail-open — @vitallium approved the bounded version in https://gitlab.com/gitlab-org/gitlab/-/work_items/623420#note_3780077140
  • Prerequisite for shipping the paid experience to self-managed in 19.4

Everything here is still behind secrets_manager_paid_experience, which stays default-off. The fallback branch and the default timeout are also behind secrets_manager_entitlement_lkg_fallback, an ops flag that's default-enabled — an off switch back to today's behaviour on .com, and one a self-managed admin can flip too.

Choices a reviewer will probably ask about
  • The slot holds CustomersDot's two raw responses, not the merged entitlement. The merge derives the subscription grace window from the clock, so replaying the inputs re-evaluates it against today and a grace window that lapsed mid-outage stops being extended. trial.state is the exception, since the client computes it at parse time — an expiring trial keeps serving :trial for the rest of the window. That is deliberate: deriving it locally would deny a customer who has since converted to paid.
  • Gitlab::Redis::SharedState, not Rails.cache. The cache instance is eviction-eligible, and an evicted slot is indistinguishable from an expired one — it would go missing under memory pressure, which is exactly when this is needed.
  • Only a transport failure triggers the fallback, discriminated by the exception's cause. I narrowed Gitlab::HTTP::HTTP_ERRORS down to TRANSPORT_ERRORS, dropping eight classes: our own SSRF, redirect, size, header-injection and invalid-response guards, plus OpenSSL::OpenSSLError and Net::HTTPBadResponse. A wrong subscription_portal_url is not among them — a bad hostname raises SocketError and a bad port Errno::ECONNREFUSED, both of which still earn the fallback.
  • The 2s timeout is a default on .for rather than something each caller passes, so all eight untimed request-path callers are covered and a new one inherits it. Net::HTTP retries a read timeout once on a GET, so the effective ceiling is roughly twice that. .for! keeps no default — those callers pass their own budget or are conversion mutations that must reach CustomersDot.
  • 24h is the conservative end of the 24–48h range agreed with Fulfillment.
  • No circuit breaker. It is the right mechanism for the fleet-wide cost of a long outage, but Gitlab::CircuitBreaker sends a synchronous Sentry event per failure in the request thread, which is what SecretsManagement::ThrottledErrorTracking exists to avoid — so using it properly means a bespoke breaker class plus its own metrics and flag, as Gitaly did. Deferred rather than overlooked.
  • Each for! caller's existing timeout is now also its replay threshold. EE::Ci::RegisterJobService (0.25s) and SecretsManagement::BillableEvents::SecretsReadEmitter (ENTITLEMENT_TIMEOUT_SECONDS = 0.25, no cache_ttl) are both low enough that any CustomersDot reply slower than 250ms becomes a transport failure and an lkg_stale serve, so the new counter will be non-zero at baseline on those two paths. For SecretsReadEmitter that changes what a billable event means: before this MR a timeout dropped the entitlement keys from the event, now a usable slot stamps it with the last-known-good state. The client never caches failures, so a persistently slow CustomersDot means every secret read makes a live call and logs one lkg_stale line. SecretsManagement::BillableEvents::SecretsStoredEmitter passes no timeout at all, so it runs on the Gitlab::HTTP defaults — a separate gap, not introduced here. No correctness impact on any of the three; all fail open. Whether to raise the two 0.25s budgets is a follow-up.

References

How to set up and validate locally

  1. Enable the flag and pick a top-level group:

    Feature.enable(:secrets_manager_paid_experience)

  2. Resolve once against a reachable CustomersDot so the slot gets written, then confirm it exists:

    group = Group.find_by_full_path('<your-root-group>')
SecretsManagement::Entitlement.for(group).state
SecretsManagement::Entitlement::LastKnownGoodStore.read(
  SecretsManagement::Entitlement::Resolver.cache_key_for(group)
)

  3. Point subscription_portal_url at an address that black-holes, or stub the client to raise a Net::ReadTimeout-caused error, then resolve again. It should return the previous state rather than :ineligible, and log source: lkg_stale with an lkg_age_s.

  4. Travel past 24h (or LastKnownGoodStore.delete) and resolve again — now it fails closed to :ineligible with source: fail_closed.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jayakrishnan Mallissery

Merge request reports