Org Mover internal API: use a dedicated secret and audience instead of the gitlab-shell token

Summary

The internal Org Mover API (/api/v4/internal/org_mover/, introduced in !253143 (merged)) authenticates with the gitlab-shell shared secret via authenticate_by_gitlab_shell_token!. Gitlab::Shell.verify_api_request only checks the JWT signature and iss == "gitlab-shell"; there is no audience (aud) claim scoping the token to Org Mover.

This means the Org Mover credential can authenticate to every other gitlab-shell-backed internal endpoint, and any holder of the gitlab-shell secret can call the Org Mover endpoints. This is acceptable for the initial iteration but should be tightened before the endpoints are relied on in production.

Proposal

Give the Org Mover internal API its own authentication, scoped to it alone. Options to evaluate:

  • A dedicated shared secret for Org Mover (separate from the gitlab-shell secret).
  • An audience (aud) claim on the JWT verified by the endpoint, following the KAS pattern (Gitlab::Kas.verify_api_request), so a token minted for one internal consumer cannot be replayed against another.

Credential design guidance (from AppSec review)

From the AppSec review of !253143 (merged) (note):

When the dedicated credential is built, an asymmetric key pair (for example ES256) is a better fit than another shared secret as it would reduce the blast radius in case of compromise.

With a shared secret, every process that verifies a token can also mint one. For Org Mover that means every Puma and Sidekiq process on every cell holds a key that can drive maintenance on that cell, and if the same secret is used across cells, on every cell. A compromised cell then becomes a way to attack other cells through this API, while cells should be isolated where possible. Per-cell shared secrets avoid that, but then the control plane has to store and rotate one secret per cell.

With an asymmetric key, only the control plane holds the private key and cells hold a public key instead. The public key is not sensitive and doesn't need to be treated as a secret. As a result, a compromised cell cannot mint tokens for any other cell. Rotation is also simpler, because the Rails side can trust several public keys at once during a changeover.

The Rails side already supports this: Mail Room verifies with public key files, and the CDot internal API with a key set plus issuer, subject, and audience checks. Org Mover could copy that pattern. Please add an audience and issuer check either way, but note that on a shared secret those are guardrails against misconfiguration, not a security boundary. Anyone holding the secret can set any claim.

Acceptance criteria

  • Org Mover endpoints no longer authenticate solely with the shared gitlab-shell secret.
  • A token valid for Org Mover cannot authenticate to unrelated internal endpoints (and vice versa).
  • AppSec review of the chosen approach.

References

  • Introduced in: !253143 (merged)
  • Auth entry point: lib/api/internal/org_mover.rb, authenticate_by_gitlab_shell_token! (lib/api/helpers.rb), Gitlab::Shell.verify_api_request (lib/gitlab/shell.rb)
  • Prior art: KAS audience-scoped verification (Gitlab::Kas.verify_api_request)
  • Part of gitlab-org#20404
Edited by Abdul Wadood