How can we solve child/parent pipelines?

Problem Statement

As pipelines grow more complex, a few related problems start to emerge:

• The staged structure where all steps in a stage must be completed before the first job in next stage begins causes arbitrary waits, slowing things down
• Configuration for the single global pipeline becomes very long and convoluted, making it hard to understand
• Imports exacerbate the above item, and create the potential for namespace collisions where jobs are unintentionally duplicated
• Pipeline UX can become unwieldy with so many jobs and stages to work with

Need statement: The user needs a way to let jobs not unnecessarily depend on other jobs within the same pipeline context so that they can let parallel job trails continue without being blocked by each other.

But, is this really complete? And what's the solution?

See also #16094 (closed)

Reach

Impact

Confidence

Effort