Define standard event schema for "fat" events

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem to solve

  • The majority of events published to the EventStore are "thin" events – they contain only references that subscribers can use to fetch more data via the API.
  • This pattern can lead to unpredictable DB load as subscribers call back into the API for information they need to complete their work.

Proposal

  • Transition to "fat" events for EventStore to ensure subscribers have the information they need to complete their work
  • Rely on existing payloads defined for GitLab webhooks to avoid adding a new source of truth for event data
  • Rely on industry conventions for shaping event schemas when possible to maximize compatibility with other systems
  • Include standard metadata within the event envelope whenever possible:
    • Timestamp for when the event occurred
    • Project associated with the event
    • User that triggered the event (if any)
    • How the event was triggered (e.g. user action, system action)

Example

Suppose we had an event that represented a new vulnerability finding.

Thin event

This is the approach that most of the events in the EventStore take: it only contains the reference to the new vulnerability object with no data about the object at all. Subscribers must call back to the API to fetch more data to do anything useful with this event.

{
  "vulnerability_id": 541
}

Fat event

This is the proposed approach: the event provides essential details about the vulnerability so consumers can immediately perform the work they need. This schema also follows cloudevents.io specification for broader compatibility with other systems.

{
  "specversion": "1.0",
  "type": "vulnerabilities.new_finding",
  "source": "https://127.0.0.1:3000/toolbox/gitlab-smoke-tests/-/security/vulnerabilities",
  "subject": "gid://gitlab/Vulnerability/541",
  "id": "94CCCB18-51C0-4D39-8EBA-DDC376BCAFF0",
  "time": "2018-04-05T17:31:00Z",
  "authtype": "app_user",
  "authid": "gid://gitlab/User/123",
  "project": "gid://gitlab/Project/392359",
  "datacontenttype": "application/json",
  "dataschema": "https://schemas.gitlab.com/17.5/vulnerabilities/new_finding.json",
  "data": {
    "url": "http://127.0.0.1:3000/toolbox/gitlab-smoke-tests/-/security/vulnerabilities/541",
    "title": "Test finding",
    "state": "detected",
    "severity": "low",
    "severity_overridden": false,
    "identifiers": [
      {
        "name": "CVE-1",
        "external_id": "1",
        "external_type": "CVE",
        "url": "http://127.0.0.1:3000/toolbox/gitlab-smoke-tests/-/security/vulnerabilities/541"
      }
    ],
    "report_type": "sast",
    "confidence": "ignore",
    "confidence_overridden": false,
    "dismissed_at": null,
    "dismissed_by_id": null
  }
}

References

Edited by 🤖 GitLab Bot 🤖