Reducing complexity in .gitlab-ci.yml files
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem to solve
Gitlab is an awesome tool to automate tasks. It allows you to define a build pipeline, but also automate infra tasks using many when: manual blocks, or schedule/trigger with only: scheduled/triggers.
Unfortunately, because it's usages are getting so diverse, sometimes they get contradictory against each others responsibility. For example, I don't want my build pipeline to start at the same time that I want run some automated nightly checks from a schedule.
I understand that moving things around in the yaml file might be hard on backwards compatibility scenario's.
Further details
Usecase: A pipeline to build a regular java application. Ops tasks for the same applications:
- scheduled nightly backup
- deploy to separate environments
- ...
Proposal
Since putting everything in one gitlab-ci file with multiple conflicting concerns is getting harder, allow a mechanism to separate them.
| file | purpose |
|---|---|
| .gitlab-ci.yml | Contains build related things, building, testing, deploying artifacts (maven, docker, nuget, npm, ...), ... |
| .gitlab-ops.yml | Contains a set of task regarding operations, backups, healthchecks, etc |
| .gitlab-deploy.yml | After artifacts are deployed they can be deployed to environments, this is where we'd start to see kubernetes/aws/etc related things |
| .gitlab-chat.yml | Chatop specific tasks |
| .gitlab-triggers.yml | Trigger specific task |
If things would need to be reused between them, they still can be, bash/kscript files.
What does success look like, and how can we measure that?
Taking autodevops as an example would be split into multiple files, but each with reduced complexity and a more SRP (single responsibility) approach. The implementations becomes more composable as well, making updating one aspect easier.
A followup change might even include .gitlab-deploy-kubernetes.yml and .gitlab-deploy.aws.yml in order to have 2 ways of deploying your application that do not leak complexity into each other's file.
What we'd definitely like to see is not having plenty of env variables used as a specific criteria to trigger certain jobs. And certain things like only: triggers/schedules should be ommitable when adding jobs in -triggers or -schedules.
I believe that splitting/supporting this way of working might be helpful to stay backwards compatible and documentation can be straightforward by specifying what attributes certain jobs would get based on the filename pattern.
A side effect might also be that it becomes easier to compete with tools like atomist. Trying to do similar things, like chatops, now often leads to unmanageable complexities within the file.
Links / references
https://gitlab.com/gitlab-org/gitlab-ce/issues/18157 https://gitlab.com/gitlab-org/gitlab-ce/issues/28592