馃挕 Workflow Automation Triggering Mechanism - Discovery
Context
To run a workflow automation, the first step is to select a trigger point, for example, when an GitLab issue is created or updated run workflow ABC.
This issue aims to compare the options and their trade-offs to select the most appropriate option based on the below consideration:
- The selected approach should align with Incubation Engineering's development guidance - "...focus should be to move fast, ship, get feedback, and iterate"
- It should not introduce reliability or performance issues to existing features.
- The goal is not to introduce a superior event-driven architecture, e.g. adding Kafka, SQS etc.
Options
web_hooks table and STI
A) Piggy-back on theThe web_hooks table is behind the WebHook model, which is inherited by SystemHook, ProjectHook, ServiceHook and GroupHook. These hooks are executed after changes are made to the models, e.g. Project::execute_hooks. The supported hooks are sufficient for the automation use case, at least for now.
Pros:
- Existing DB table and STI means no work on DB is required.
- No need to create a new model means less code to maintain.
- There is potential to reuse webhook's logging facility
- Reusing the event subscription pattern means consistency and maintainability.
Cons:
- Cardinality limits on the numbers of web-hooks for groups and projects (see https://docs.gitlab.com/ee/administration/instance_limits.html#number-of-webhooks)
- Executing automation locally does not match the webhook's original purpose - delivering data to external recipients. However, we do have an existing local use case - FileHook that fires off
SystemHook
. - All the
WebHook
's STI classes require a URL field, which is unnecessary for Automation.
B) Extract WebHook's event subscription into its own model
erDiagram
EventSubscription ||..|| WebHook : "belongs to a"
EventSubscription ||..|| AutomationHook : "belongs to a"
EventSubscription ||..|| FileHook : "belongs to a"
In this solution, the EventSubscription
model and the EventSubscriptionService
become the middle layer to decouple the entities from the reactions (hooks).
Pros:
- Cleaner separation of concerns
- The decoupled of event publisher and consumer through the subscription model makes it easier to support Pub/Sub architecture in the future
Cons:
- The refactor touches the core product feature that requires delicate implementation that may slow down the iteration of the Workflow Automation
- The indirectness is likely to introduce more DB queries that impact performance
- The approach may not align with the event-driven architecture blueprint
C) Create a dedicated configuration table for triggering workflows
This solution was part of @grzesiek's GitLab CI Workflows PoC. The ci_workflow_events is a dedicated table to support a more granular event subscription for executing CI based workflows.
Pros:
- The dedicated event subscription table means more opportunities to optimize for the workflow automation
- The design is more extensible means supporting new event type should be easy
Cons:
- Duplicated event subscription mechanism
- It requires effort to add DB table, model add attaching to entities before delivering customer value
Conclusion
As discussed, option A is chosen to unblock the first release of Automation that focuses on UX. We've also agreed to migrate Automation to the new eventing architecture from CI Workflows when it becomes available.