Use asymetric authentication in cells mailroom requests

Goal

Replace the symmetric (HS256) JWT authentication used for the cells mailroom service's requests to the internal mail_room API with an asymmetric algorithm (RS256/ES256), so the signer holds a private key and each cell only holds a public verification key.

This is required for the cells email-ingestion architecture: a single mailroom service forwards incoming email to the owning cell's internal mail_room endpoint, and that request must be authenticated without sharing a forge-capable secret across cells.

Current state

  • The mailroom service authenticates to POST /api/v4/internal/mail_room/:mailbox_type with a JWT, verified by Gitlab::MailRoom::Authenticator via Gitlab::JwtAuthenticatable#decode_jwt.
  • The algorithm is HS256 (symmetric): the mailroom service signs with a shared secret and Rails verifies with the same secret (incoming_email.secret_file / service_desk_email.secret_file).
  • Because the secret is symmetric, anyone able to verify a token can also forge one. In a cells topology every cell would need that same secret, so a leak from any single cell would let an attacker forge mailroom tokens against every cell's ingestion endpoint — turning a single-cell incident into a cluster-wide email-injection one. This conflicts with the cells principle that cells are isolated trust boundaries with no shared runtime trust.
  • Scope of the risk is limited to the mail_room ingestion endpoint (submit raw email: reply-as-user, issue/note creation, Service Desk); it does not grant general API/repo/admin access.

Proposed change

Move mailroom token authentication to an asymmetric algorithm:

  • The mailroom service holds the private key and signs the JWT; each cell holds only the public key and verifies. A leaked public key is harmless (it can verify but not forge), so a single-cell compromise cannot forge tokens against any cell.
  • Use RS256 (or ES256). RS256 already has tooling in the codebase (Authn::Tokens::Jwt, Gitlab::Ci::JwtBase, used by CI job tokens), so the verification side is a small lift.
  • Add the asymmetric verification path via a new internal endpoint / code path rather than making the existing endpoint accept both algorithms, to avoid algorithm-confusion vulnerabilities.
  • Support key rotation: include a kid in the JWT header and let cells hold multiple valid public keys at once, so keys can be rolled without a hard cutover. The mailroom signer should likewise support a rotation of private keys.

Considerations

  • Gitlab::JwtAuthenticatable#decode_jwt and Gitlab::MailRoom::Authenticator are shared by other internal clients (Workhorse, KAS, Pages, Knowledge Graph), so any change must be additive and must not alter the existing HS256 behavior those clients rely on.
  • Key distribution: the public key(s) must be provisioned to every cell; the private key only to the mailroom signer. Coordinate the deployment mechanics (and rotation) with Tenant Scale / Delivery.
  • Requires AppSec review (@gitlab-com/gl-security/appsec).

References

  • Architecture discussion: gitlab-org/gitlab&22510 (AppSec recommendation for per-cell, asymmetric, rotatable keys).
  • Related cells secrets-isolation issue: gitlab-org/gitlab#448305.
Edited by 🤖 GitLab Bot 🤖