Skip to content

Dynamic child pipelines

Lukas Eipert requested to merge leipert-build-orchestrator into main

Utilize dynamic child pipelines

This is the first big step in a bigger refactor. Dynamic child pipelines will allow this project to be way more flexible.

In the moment it basically does the following: Iterate through all the configs and parse their .tool-versions from the fork (if they exist). If not, fall back to the version from the GDK (safe default?) Create a dependency graph of what dependencies (e.g. node, golang and ruby) each config. Based on that we can generate:

  1. A job to build the "base" image. It is the same job as before, but executed in the child pipeline.
  2. Dummy jobs for each tooling dependency (e.g. node-18). They do nothing, but optionally depend on the base image job. In a future iteration they would add the tools provided.
  3. Real jobs for each config file. Those optionally depend on 2.). Under the hood they still use the trusty base image we currently use for renovate.

The biggest change in this MR is that we generate one CI job per renovate config instead of grouping them by folders. But it already puts infrastructure in place in order to:

  1. Only build the base image when we need to (e.g. the SHA of the files inside bot_image changed)
  2. Dynamically build dependency images, e.g. if a project bumps their nodejs version in .tool-version, we would dynamically generate a new job building the image
  3. In MRs: Only run jobs if the corresponding config changed.

Get full image matrix

Generate docker files based on config

Split orchestration script into smaller, more logical chunks

Add docs for the dynamic build matrix

Edited by Lukas Eipert

Merge request reports