Skip to content
GitLab
Next
Closed
Issue created by Mark Pundsack@markpundsackContributor

Grand unification theory of CI/CD

I'm trying to capture some notes on a discussion I had with @ayufan. These might not make complete sense without context.

Here's our raw notes (which will make almost no sense):

And my attempt at elaboration before I forget:

We have a long outstanding issue that getting started with CI/CD can be kind of tough when it comes to understanding caching and artifacts. And we have other issues that pipelines doesn't go far enough, and that people really need directed acyclic graphs (DAGs). Assuming we want to support these use cases, we could think about ways to have different modes for CI to operate in. For example, CodeShip has a separate produce for single vs parallel execution. I don't want to do that for a variety of reasons, not the least of which is that our ease of doing parallel builds is a huge differentiator. But it probably means they have an easier onboarding experience.

But another way to think about it is, can we introduce new, simple primitives that just default to the current way of working, but if you change the defaults, result in these other use cases.

  • Workspaces => simple

Recently CircleCI announced their workspaces feature, which aims to solve the same problem. I don't know how their feature works with parallel builds, but assuming there's a solution, it seems really promising. Basically, you declare which jobs share a workspace, and we make sure they share a filesystem so you don't have to worry about passing file changes between jobs.

Maybe we even start by defaulting to every job sharing the same workspace, although that will obviously break current pipelines. So starting by assuming everything is independent, but letting people specify a global workspace is nearly as easy as specifying some kind of mode=simple. It's unfortunate that it can't be default, but if all of our starter examples declare a global workspace, it'll catch on quickly enough.

  • Intra and inter stage dependencies after => advanced

We have a great pipeline flow with stages, but what if we let you declare a dependency at a finer granularity than a stage? Like one job could depend on another job within the same stage. Or one job could depend on a specific job in the previous stage so it can start as soon as that job finishes, without having to wait for the entire stage to finish. This might sound overly complicated, but when you're trying to get your CI/CD pipeline down to 5 minutes to improve developer productivity, this kind of control matters.

Note that flow dependencies are distinct from artifact dependencies. You may build a release artifact in a build stage, then test it in a test stage, and finally deploy it in a deploy stage. That final deploy job has an artifact dependency on the build job, but a flow dependency on the test jobs.

Now, by default, the flow control dependency is the previous stage, but if you declare an explicit dependency, perhaps with an after keyword, you can have finer control.

This is an alternate way of handling things, that isn't as flexible and I really only included to grab it's ascii art.

  • Docs ⟼⬤⟶⬤ Android ⟼⬤⟶⬤⟶⬤ iOS ⟼⬤⟶⬤⟶⬤

Imagine a common scenario where a monorepo has both an Android and iOS app in it, as well as documentation. You might want to declare parallel pipelines like:

Docs     ⟼⬤⟶⬤
Android  ⟼⬤⟶⬤⟶⬤
iOS      ⟼⬤⟶⬤⟶⬤

But with an explicit way to skip jobs based on files modified, you could represent this as a single pipeline

Docs     Android      iOS
⟼⬤⟶⬤ ⟼⬤⟶⬤⟶⬤ ⟼⬤⟶⬤⟶⬤

In current GitLab CI/CD, this will even work, but it will evaluate the various steps sequentially. So now imagine that the first Android stage declares a blank after, then it will kick off immediately (if any Android files have changed). Likewise for iOS. So we'd still represent it as a pipeline, but where execution isn't limited to a single state at a time. When looking at the mini graph or full pipeline graph, we'd just have multiple jobs running simultaneously.

  • Analogy: 3D => 2D projection (in our case, 2D=>1D)

Long discussion about the challenges and values of the pipeline mini-graph. We could show a complete DAG once you click to see a pipeline, but when representing a pipeline on other pages, like the Merge Request Widget, it's really valuable having the mini-graph. But, rather than killing the view completely, we can think of a projection, an imperfect representation of complicated data in fewer dimensions. The value of the mini-graph is quickly conveying status/progress, and navigation. And it turns out, those still work. What it doesn't aid in is comprehension of the entire pipeline, but you can click through for that.

  • Flow control

So now that we're discussing graphs, it's worth considering how we can clean up flow control (jobs that only run based on success/failure of previous jobs/stages). It would be nice to get rid of the awkward flow control we currently have, and at least graph it better, but possibly enhance it. Like be able to run a job on failure of another specific job. e.g. today, a poorly constructed pipeline may send a slack notification when tests pass, but then if that notification fails, might try to send another slack notification about the (wrong) failure. We can't tell exactly which job/stage we're talking about.

  • External/manual/stageless - Branch delete => close review app

Likewise, we have some jobs that really are outside the pipeline, like deleting a review app when the branch is deleted. Sure, you likely also make it a manual job so it has some place in the pipeline, but it's reasonable to want it to only fire when the branch is deleted. We've talked about making them stageless so they can be visually represented outside the pipeline.

Similarly, I've started declaring helper functions in my pipeline that really have nothing to do with the specific pipeline at all, but are conveniences for me because they use the project variables, are easy to trigger, and have some (inadequate) logging.

It would be nice to think of these functions as outside the specific pipeline run. If I trigger then, I think I'd rather see a new pipeline generated so it stands out in the history. There's a parallel there when thinking about ChatOps. You can define all your ChatOps custom functions in your pipeline as external tasks.

  • DAG with multiple entry points

So, what we end up with is a DAG with multiple entry points, that, along with reasonable defaults, simplifies down to pipelines, or even to single-runner sequential.