We would like to learn about the template design of the interpolation ($[[ inputs.the_input_id ]]) whether we made a good decision to follow the YAML-level interpolation (based on parsed structure) instead of text-level interpolation (where we process text first, and parse it next).
Without this feature I would need to know the exact job names that will be injected by the component (breaking the encapsulation principle) and override them in the main yaml:
include:-component:gitlab.com/gitlab-org/dast@1.0dast-api-scan:rules:-if:$CI_COMMIT_BRANCH =~ /feature-*/-if:$CI_COMMIT_TAGdast-some-other-scan:# repeat for all included jobsrules:-if:$CI_COMMIT_BRANCH =~ /feature-*/-if:$CI_COMMIT_TAG
I wasn't sure which of those issues to followup in, so I'll just reiterate what I said in #384913 (comment 1308034925) that I don't think we need all of these separate constructs. We only need two things:
steps
workflows (i.e. reusable/nested/child pipelines with stricter semantics)
Both constructs can be declared on jobs explicitly and we can avoid the use of includes entirely for this purpose. Introducing new syntax (i.e. uses) makes sense here because (1) it is functionally different from how includes behaves and (2) it unifies the two features syntactically:
I agree that we need steps and workflows, but I also think that we can't simply abandon templates and leave them as they are today. Templates will not go away - too many people depend on them. All reusable configuration files for GitLab CI are templates right now, steps and workflows do not exist yet.
What I think we should do first is to make templates (with interpolation) publishable in a catalog, after someone just runs find ./ -name '*.yml' -exec sh -c "cat spec >> {}" \; to bulk append / prepend a spec for them, but I also understand why we want components with the directory structure (one component per directory). I would prefer to start with templates with the interpolation being published using deployment manifests, but it might be more work than publishing components /cc @marshall007@fabiopitino
both templates and workflows will support <job>.steps and <job>.uses, but similar to trigger it will be mutually-exclusive with most other job-level keywords
templates can reference workflows with <job>.uses, but workflows cannot reference templates because include is not supported
templates can be converted to stricter worklflow syntax over time
Yes, I agree that this would be an interesting possibility to implement a language that is a subset of what we are supporting right now for .gitlab-ci.yml.
Another solution is to just support everything we support today, but to make it necessary to trigger a workflow within a completely new pipeline context: without merging configurations.
I think that we would need to think a lot about how we want workflows to look like. We need a separate issue and presumably a separate subpage in our blueprint about this. This is not a trivial problem to solve, and perhaps we should spend some time on a research.
All that being said, I think that we should probably start with steps instead of workflows. WDYT @marshall007?
@grzesiek I put this table summary together because it was becoming difficult for me to keep all the possible configurations straight in my head. Please feel free to copy this into a blueprint sub-page or separate issue, whatever is appropriate.
I'm not sure where it makes sense to start. I feel that we should have specs for the entire system loosely pinned down before we start on either implementation because getting the interfaces between them right is critically important.
If you look at the list of job-level keywords, I think the limitations on jobs that use steps (even in a traditional template) will need to be almost the same as any jobs within formal workflows.
@marshall007 - I found the table a useful way to understand how a workflow might work. Maybe a good next step would be to start an MR for the section/subpage of the blueprint that would cover the workflow concept? That could be a good place to iterate on the concept and add artifacts like this table.
We should probably also update the Glossary terminology to include template and workflow. WDYT?
Problem: As a user of components I would like to specify where in the pipeline the jobs need to be injected
Today we allow to interpolate stage: value with input parameters but we cannot support needs: with interpolation because we don't support anything that is not a string.
As a user I would like to specify where in the DAG the jobs need to be added. Since the value of needs: is not something the component should be aware of, I would like to inject it from the outside:
I've moved to templates just because I imagined that it would be possible to pass array and dynamically create job needs. It can provide a way to reconnect jobs in manner you described as DAG in docs. As always gitlab punished me because passing array is not possible.
I want just to mention that you are writing software in not serious way: you are not trying to implement a complete feature, just writing some stuff that may be connected to this feature.
Consequently your milestones has no features, it has just a bunch of stuff that may help someone. Nobody is targeting features, so features just never happens.