Discussion: Simulate your pipeline with different conditions and inputs before you run it
🧩 Problem
Creating a complex pipeline configuration that is as efficient and reliable as it can be will require iteration. Organizations are interested in tools that can help them reduce the time to an efficient and reliable pipeline.
To solve for this you want to reduce the length of the iteration, and the number of the iteration cycles that it takes you to get the pipeline to a good state, which reduces the total feedback loop. You don't want to have to constantly break things as you're figuring out the kinks, so ideally you get things working smoothly in a couple commits (of course it never happens like this, but it's the dream).
Things that contribute to how slow or fast is your iteration+learning feedback loop are:
- The pipeline runs slowly for one of the many reasons
- The errors you get are not easily actionable
- The syntax is confusing and difficult to learn
- The documentation is difficult to navigate
Addressing the errors, warnings and the different processes in the pipeline takes significant time on its own, plus every time you want for the pipeline to run, and things break which slows the organization down.
👁 ✨ Potential solution
Preventing the failure and inefficiencies could really help avoid all this pain. Even small improvements in failure prevention can have a great impact on the perceived usability and ease of learning.
Testing your pipeline in different conditions and with different inputs before running the pipeline could really help you improve your pipeline before it's used by the team.
Testing pipelines with different conditions and inputs
- Is it a Merge Request pipeline or pipeline based on specific file changes?
- What Branch is it for?
- What variables has it consumed?
Simulating failed jobs
You could also figure out lots of kinks in your pipeline if you could test a pipeline with some jobs set as "failed", to see how your pipeline behaves when one of the jobs fails. You can then create a more reliable architecture with the insights from such test.
We could offer a way to do this in the pipeline editor and if possible locally for those who use their editor of choice.
❓ Questions
- Currently we have the "simulate pipeline" option in the Linter. What value does it give you? We don't communicate it in the UI.
- How can we take inputs and set different conditions on the test?
- How can we simulate the most realistic test without actually running the pipeline?
Further details
When authoring a pipeline, engineers want to be able to test a pipeline in different contexts to test and optimize their pipeline for realistic scenarios.