Architecture Decision: Where do we sign SLSA provenance?

Context

As part of #514814 (closed) we're implementing a CI/CD component that produces SLSA provenance attestations. It signs provenance statements using Sigstore cosign.

We want to explore other options, and select the best one.

Note: The best component to generate provenance statements before signing them is discussed in Architectural Decision: Where do we generate SL... (#537049 - closed).

Options

SLSA provenance statements could be signed in the following components:

• runner
• GitLab backend
• a new signing service

Note: This list can be updated with other options identified when solving this issue.

Criteria

Requirements

• can receive provenance statement - see Architectural Decision: Where do we generate SL... (#537049 - closed)
• can safely handle private keys
• can connect to GitLab Secrets Manager
• can connect to external KMS
• meets SLSA requirements for L3
  • Is it in the control plane?
• can connect to Fulcio
• can send (signed) provenance attestation so that we can upload it as an artifact

Aspects to be considered when comparing options:

• infrastructure complexity
• maintenance complexity
• effort to integrate Cosign
• development effort
• familiarity with component and its tech stack
• alignment with architecture
• security boundary
• scalability (computational cost is relatively low)

Tasks

• Evaluate options (discuss in issue)
• Make decision (discuss in issue)
• Capture context, options, decision, and consequence in ADR (open MR)
• Update design doc with decision, and link it to ADR (open MR)

Decision needs to be added to https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/slsa_level_3/#decisions

/cc @darbyfrey