We recently introduced dynamic child pipelines which allows for generating a .gitlab-ci.yml at runtime and starting a pipeline based on the generated YAML. However, we don't provide good examples for how this can be done and what other kinds of tools are out there that can make this easy.
Add a .gitlab-ci.yml or project template that demonstrates a competent implementation of this. Project template is probably more realistic as there will be supporting template files.
@ayufan is there any reason not to use what you build there as the MVC of our template? One thing that comes to mind is maybe show an example of a function, variable, and loop. On the other hand simple is nice too, and we can just link to the Jsonnet reference.
This is really slick @ayufan. It sticks with a standard DevOps language, while at the same time not being completely static. Meanwhile, it has far better readability (Just my opinion), than many solutions.
Is the idea that we'd extend this to enable utilization of the ENV vars of a job? I go back to the monorepo for the base use-case: N subdirs, each of which is its own product. When the job runs, I need to look at the GIT changes & generate the list of subdirs to build. Then I loop through this dynamically generated list & call the downstream for each.
I think this is a great use case, as it is a common occurrence & when a change to the parent build script logic can break everyone, many orgs don't want each dev team modifying the parent job file.
Additionally, even the author discourages running jsonnet on a server side unless using that in a forking model where we can strictly control process lifecycle: https://jsonnet.org/ref/bindings.html.
But, for anything more complex we would need to add a proper simple maybe docker image to generate this config and have a "tool with functions" predefined.
Yeah, I think the problem here is that it eventually falls back on the "logic" supported by our existing YAML solution. As long as we fall back on the YAML logic, we should stop trying to say YAML must remain static, and treat it as Ansible has.
The same as when architecting an application: there should be a very clear requirement / distinguishing purpose for each section of the CI "language". EX: Jsonnet has logic & generates a static YAML and GitLab executes the static YAML. Then I know to use jsonnet-lint to validate my logic scripts ci/lint to validate my static YAML. I know to use _____ IDE to step through my code & ____ for Unit testing. EX: I can download the source code of Ansible and use PyCharm to debug complex problems when executing or testing Ansible. As long as there is a mixture of languages & purpose in a single file (our current YAML solution), there will be problems in scalability.
We've had conversations with @andrewn about jsonnet for the metrics dashboard yml, I think it'd be good to involve him in this conversation as he has managed the jsonnet setup for the grafana dashboards for the infrastructure team
@jyavorska Personally I think it's important to raise visibility about an issue being worked on in a given milestone. I thought it would be helpful to see this existed while it's in flight, but I'm also happy to be cc'd on it next time. I do verify all the Release posts after all :).
As for the respective workflow labels, probably could have just been ~"workflow::In dev" -> ~"workflow::In review" -> workflowproduction, but I don't have a strong opinion on this here. Raising awareness was the main intention of my original question.
Thanks @cheryl.li, I had thought that production was only applied by the bot when it is actually in production. This particular MR spent about 3 hours between starting development and being reviewed.
@jyavorska True, I think just awareness prior to a Release post MR is enough for me. And if it's a short dev cycle, the workflow labels don't make a ton of sense as you said.
I am trying to use jsonnet to generate a pipeline, but:
not so severe issues:
in your demo, you generate JSON, but name it as .yml file -- it seems to work however
renaming the file to .json, GitLab refuses to use it since it is not YAML (based on extension?)
using Jsonnet's std.manifestYamlDoc/1 and invoking jsonnet -S, I can get a YAML generated, which is OK but probably not required as long as GitLab can consume JSON if the filename is .y(a)ml
these are actually blocking me from finishing what I am trying to do:
it is challenging to generate things like entrypoint: [""] or rules section with various if conditions in there -- the escaping gets broken mostly and one has to work around these issues
for this, or another reason unknown to me, the jobs having rules are not executed, even if they are shown in the CI linter for the same generated output
the pipeline trigger, despite the generated output passing the CI linter, is failing with an "unknown failure" and the child pipeline is skipped. There is no error, no information on what is wrong.
other notes:
since jsonnet can check for various CI variables to control what pipeline gets generated, can it also check for things like changed -- would it be worth making the list of changed files available somehow?