Serve Secrets Manager last-known-good entitlement on CDot 5xx

Dmytro Biryukovrequested to merge
629156-lkg-fallback-covers-cdot-5xx into master

What does this MR do and why?

The Secrets Manager entitlement resolver calls CustomersDot (CDot) to decide whether a customer can use Secrets Manager. A last-known-good (LKG) fallback added in !254445 (merged) replays CDot's last successful answer when CDot cannot be reached, but it only fired on transport errors. A CDot HTTP 5xx was raised as a plain status error with no cause, so the fallback never fired and the resolution failed closed to ineligible, hiding the sidebar, 404ing the secrets page, and failing CI jobs. Since 2026-09-01 this happened at least 44,240 times, about 97% of them HTTP 502 from CDot's nginx while its Puma workers were wedged.

This MR makes 5xx responses eligible for the LKG fallback:

  • Adds a Gitlab::SubscriptionPortal::SecretsManagerResponseError base class with cdot_status and cdot_block_reason, and an UNAVAILABLE_STATUSES list of 500, 502, 503, 504.
  • Has both per-endpoint Error classes inherit from it and adds an UnavailableError subclass for each. Callers that rescue the base Error are unaffected.
  • Makes the REST client raise UnavailableError for 500/502/503/504 and a plain Error for any other unexpected status, carrying the HTTP status and any block_reason from the body, and handling HTML or unparseable bodies.
  • Makes the resolver's fallback treat UnavailableError the same as a transport error. 402 still maps to a blocked response and every other 4xx still fails closed.
  • Adds cdot_status and cdot_block_reason to the fail-closed and stale-served log lines.
  • Updates the Prometheus metrics doc so lkg_stale is documented as covering 5xx as well as unreachability.

The existing secrets_manager_entitlement_lkg_fallback kill switch still gates the whole fallback, including the new 5xx path.

Out of scope: alerting on the fail_closed counter and fixing CDot's own capacity issue.

References

Screenshots or screen recordings

No UI change.

Before After

How to set up and validate locally

It is not practical to make a real CDot instance return a 5xx from GDK, so validate with specs or a console stub.

  1. Run the updated spec files:

    bundle exec rspec ee/spec/lib/secrets_management/entitlement/resolver_spec.rb \
  ee/spec/lib/gitlab/subscription_portal/clients/rest_spec.rb \
  ee/spec/lib/gitlab/subscription_portal/secrets_manager_response_error_spec.rb

  2. Or, in a Rails console on a SaaS-simulating GDK, stub Gitlab::HTTP.get to return a double with code: 502.

  3. Call SecretsManagement::Entitlement.for(group):

    • after one prior successful resolution, it returns the stored (warm) state.
    • with no stored state, it returns ineligible.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports