Draft: PoC NPM CI events

Disclaimer: The following contains information related to upcoming products, features, and functionality.

It is important to note that the information presented is for informational purposes only. Please do not rely on this information for purchasing or planning purposes.

As with all projects, the items mentioned in this document and linked pages are subject to change or delay. The development, release and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

🌳 Context

This is a Proof-of-Concept that comes from a merge of two other Proof-of-Concepts:

1. Draft POC CI Pipelines for packages (!63420 - closed). Video presentation.
2. Draft: GitLab CI Workflows PoC (!91244 - closed). Video presentation.

(1.) is presenting the idea and the possibilities offered by starting a CI pipeline when a package is published.

(2.) is presenting the idea of triggering a CI pipeline from a system event, such as creating an issue.

This MR presents a mix of both changes to:

• trigger pipelines from Package Registry operations using https://docs.gitlab.com/ee/development/event_store.html

🍿 Demo time!

1. pull this MR branch locally.
2. have a project ready with the following .gitlab-ci.yml file:

image: alpine:latest

package_published_event_job:
    on: "Packages::NpmPackagePublishedEvent"
    script: 'echo "Hello from package_published_event_job: $PACKAGES_NPMPACKAGEPUBLISHEDEVENT_PACKAGE_FILE_ID"'

standard_job:
    script: echo "Hello from the standard_job"

tag_created_job:
    on: "Packages::NpmTagAddedEvent"
    script: 'echo "Hello from tag_created_job: $PACKAGES_NPMTAGADDEDEVENT_TAG"'

tag_deleted_job:
    on: "Packages::NpmTagDeletedEvent"
    script: 'echo "Hello from tag_deleted_job: $PACKAGES_NPMTAGDELETEDEVENT_TAG"'

From here, several options:

1. Publish an NPM package following these instructions.
   • A new pipeline is created and package_published_event_job is executed.
2. Create a new NPM tag.
   • A new pipeline is created and tag_created_job is executed.
3. Delete an existing NPM tag.
   • A new pipeline is created and tag_deleted_job is executed.

🔍 Main changes

Package Registry:

• Added 3 events for when an NPM package is published, a NPM tag is created and a NPM tag is deleted.
• Add a pipeline_id column to the package files table.
• Add a listener to the CI pipeline created event to link a package file with its pipeline.

CI pipelines:

• New key words for jobs: on:
• Create a new table to store events subscriptions. For each project, we're going to store which events the jobs config of that project is interested in.
• Create 2 new steps in the create pipeline service sequence:
  • subscriptions update: this one will check the current jobs of the current pipeline. If there are interested in the events, an events subscription for that project is created or updated.
  • subscriptions select: if the pipeline was triggered by an event, this one will match the event with the jobs. If the pipeline was not triggered by an event, it will remove all jobs that are interested in events.
• Add a new source type: :event_subscription.
• Add a listener to the Package Registry events. That listener is completely generic and does not depend on the event itself. It will simply:
  • Trigger a new pipeline with source :event_subscription
  • pass the event payload to the create pipeline service.
  • set pipeline variables using the event payload.
• Update the CI pipeline created event to host:
  • an event class.
  • an event payload.
• Update the create pipeline service so that when the CI pipeline created event is triggered, the original event payload and class is passed.

Event store:

• Connect the CI listener to all package registry events. Conditional dispatch: the target project must have an event subscription.
• Connect the Package Registry listener to the CI pipeline created event. Conditional dispatch: the original event must be one for the package registry.

🔮 Conclusions

• Highly decoupled changes: the package registry changes are not aware of the CI pipelines changes and vice-versa.
• High re-use of existing components between the GitLab event store itself and the CI created pipeline event.
• The pipeline creation logic must be updated to accomodate this new "source" of pipelines: events from the event store.
• The current naming might need an update. In this MR, you need to specify the exact class name in the ci config file (the on: field). It might be worth to have a mapping here so that event class names are simplified for users (Packages::NpmPackagePublishedEvent -> package_registry/npm/package_published)
  • This could open up "smart" selectors for events. Eg. have a job that has on: "package_registry/*" -> that job is executed on any package registry event.
• Possible downside: the condition dispatch is evaluated when the event is triggered.
  • This means that when we receive the API request that is a package publication, we will trigger an event and evaluate its dispatch conditions. Currently, those conditions involve a database query which means that we're adding one more query to an existing web request.
• From the previous downside: if a event trigger needs to evaluate a large amount of dispatch conditions, we might introduce a scalability issue here.

❓ Questions

• We choose to pass around the event class and its payload so that each part is free to access the payload. Is there a better way to do this?
• We choose to pass the event payload to the jobs using environment variables. Is there a better way to do this?