A common use case for includes is to want to position something at the beginning or end, but as the author of a shared include you don't necessarily know what the first or last stages will be called. This makes it difficult and error prone to try to write a job that runs at the start or end of a pipeline.
Intended users
Authors of includes
Further details
This solution came from discussions around https://gitlab.com/gitlab-org/gitlab-ee/issues/29980, which would make it possible to merge stages across includes. That ended up being a very hard problem to solve in an easy to use way, and we discovered this solution which was simpler and addressed most of the use cases. A new issue was opened because this one does not solve the exact same problem as the other.
Proposal
We will add two new pre-defined stages called .pre and .post These would exist alongside the other predefined stages, build, test, deploy, and would be sequenced as follows:
.pre
build
test
deploy
.post
.pre would always be guaranteed to be the first stage, and .:post:. would always be guaranteed to be the last.
If you write a job in your include that needs to run at the beginning, you'd insert it as follows:
my_job:stage:.pre
Any other includes that also need to run at the beginning would use this pre-defined .pre stage, and all of these would run in parallel. .post is used and works the same, but is always at the end of the pipeline.
Permissions and Security
Not applicable
Documentation
Testing
What does success look like, and how can we measure that?
@jlenny Are we okay with the fact that users will be able to circumvent this after https://gitlab.com/gitlab-org/gitlab-ee/issues/30632 is implemented by specifying a job that needs the job that's supposed to run last?
I don't think that we have any safeguards today that would even disallow you to overwrite the compliance job.
Secondly, this syntax is not allowed today, as each job needs to have stage, and needs: requires job in previous stages. If we allow the same-stage needs: then it would be a problem, but maybe this should be avoided differently, by marking some jobs as locked: true, so they could not be overwritten, and they could be needs: on?
This would actually pre-included includes to actually provide a security mechanism preventing from removing and changing compliance jobs.
That's why I originally said that this workflow will stop being viable after #30632 (closed) is implemented. You'll be able to create a set of jobs that run after the ::post stage by using DAGs.
@jlenny Does this actually solve any use case? It can't guarantee that a job will be ran last, or that it can't be overwritten, so it doesn't sound like it's solving a use case for people that want to run compliance jobs.
@mbobin @jlenny To recap - we're going to move forward with this. The primary use case isn't a security control, but more of a order of execution syntax sugar.
@mbobin We can achieve this by prepending and appending stages before processing the CI yaml. We can't do it after or during the processing because it'll be a mess to properly validate.
Let's get the ball going on implementing this - let me know if there's anything I can do to speed this up.
@jlenny I have a small problem with those names, especially with the :: because : is a special character in YAML and ::something will be parsed into a Ruby symbol :":something", not string "::something". Example:
I'm fine with changing them @mbobin, but @ayufan came up with them so please check with him in case there was some other technical reason for prefixing them.
It seems that we can use .pre. Maybe we can use :pre? But maybe using symbols is not a bad idea, as this is significantly different than strings.
So, maybe we should use :pre to indicate a special purpose. It means that we do not allow to use :pre in stages:, but we allow and transform it for stage:?
Great @mbobin. .:pre:. was the name that @ayufan ended up recommending? It looks a little strange to me, in particular with the :. at the end. Can we simplify it to .pre and .post or something like that?
@jlenny I didn't wait for his response, but from his note(#29980 (comment 215826653)) it can be anything that it's unlikely to be in use by anyone today because it might break those pipelines. I think this is the the only concern.
@mbobin that makes sense. I like something simple so .pre or .post seem fine to me. Does the dot prefix mean anything else in YAML that we should avoid? @matteeyah did you have any thoughts on anything simple but more expressive?
Great news @mbobin! Since this one is down to the wire, please @ me when verification is complete and this issue gets closed so I can merge the release post content.
@mbobin there seems to be some question now of if this actually made %12.4 - there is a new deployment to production dated Oct 20th that may include this change. Can you please validate in production (urgently, if at all possible) to see if we can include this in our communication plan for the %12.4 release?