Provide authenticity for streamed audit events

Audit events can be streamed from a GitLab instance to an independent destination. See &5925 (closed)

Problem

• Currently it is possible to spoof streamed audit events as there is no way to verify that an incoming audit event is legitimate.

Proposed Solution

• Add authentication header to outbound streamed events that the consumer can validate.
• This should be generated by the application on a per-destination basis.

Implementation Plan

• Add new field verification_token to external_audit_event_destinations table. It should be not null.
• Update GraphQL creation mutation to require the input of a token.
  • In hindsight, this won't work for existing destinations. It should be auto-generated and available to the user via the API.
• Update payload worker to send the token as part of the request headers.

Discussion before implementation was decided:

One of the open questions was:

Security requirements?

1. Do we require any sort of authentication before sending? Encrypted/cleartext? Dig in as part of implementation planning.
2. Do they need requests to be signed by GitLab (or the self-hosted instance yes)?
   1. Intuition says yes

Proposal - Interim

• When we publicly release streamed audit events, make a note in the documentation that this feature is a TODO.
• Suggest interim measures such as validating the IP address or TLS certificate presented by the sender? (Does GitLab.com use a known set of IP addresses? Self-hosted customers might have more luck here)

Questions

• Does a pattern for this exist already? I.e. a GitLab signing key? Does it exist for .com and self-hosted, or just one or the other?

Potential solutions

• A GitLab-generated private key signs messages, and the destination validates the signature using the public key
  • Pros:
    • Standard way of doing things. (Learn more: https://auth0.com/blog/how-to-explain-public-key-cryptography-digital-signatures-to-anyone/)
    • Probably easier to hide a private key and prevent arbitrary users from reading it (impersonation / spoofing)
  • Cons:
    • have to generate and securely store a private key
    • users might want to bring their own signing keys
    • keys need to be distributed to whatever parts of the stack emits the message (might be hard in a k8s cluster?...)
• A user-generated / visible simple secret is shown in the UI, and sent as a header
  • Pros:
    • Very simple
  • Cons:
    • Anyone with access to that part of the UI or the DB can see the secret (we could make it one-time visible?)