With the new needs feature in GitLab CI we can speed up our pipelines.
There are several ways to do this:
All of the allow_failure jobs in the test stage do not need to hold up deployment
build and test can run in parallel because the test job builds it's own test image anyway
We could also consider in review apps to start deployment as soon as the build is done. Since it's a review app we may be fine to deploy regardless of failing tests.
So basically I see the following additions to the CI YML:
Some of this can be done now but the needs: [] part is blocked by #30631 (closed) and that will have the biggest speedup in allowing this to kick off earlier.
Designs
An error occurred while loading designs. Please try again.
Child items
0
Show closed items
GraphQL error: The resource that you are attempting to access does not exist or you don't have permission to perform this action
This was my intuition as well but I wanted to try to gather some data about this before I made the change. It was proving difficult to get reliable benchmarks here. I'd love to see us get some good monitoring data about this stuff before/after we start using needs. If we have a way to log total time taken for Auto DevOps pipelines to Prometheus I think this would help immensely in demonstrating the impact of the changes to using needs.
Another thing I've been questioning is the whole allow_failure section. Given how easily ignored allow_failure jobs are in general, I would be curious to see usage numbers for jobs in that section
If needs: is set to point to a job that is not instantiated because of only/except rules or otherwise does not exist, the pipeline will be created with YAML error.
We are using rules, but the same applies. For example:
if VAR is undefined, then the job fails with job_1: needs 'job_0', because job_0 was never created
This means that
needs: [test] is out of the picture, because the CI variable TEST_DISABLED blocks the creation of that job
needs: [build] is a tough sell because limits the re-usability of the Jobs/Deploy.gitlab-ci.yml outside of Auto-DevOps.gitlab-ci.yml and may be undesired as it will result in higher CI minute usage for customers
Another problem with needs is that jobs in Jobs/Deploy.gitlab-ci.yml would need to know about jobs in other templates, which breaks encapsulation. For the sake of encapsulation, it would be nice to have an inverse of needs, a keyword
Thanks for clarifying that. Is this something we can take to Verify folks to see if the scheduler should start handing out jobs when no more blocking jobs are left in a stage?
I found a couple of related issues #30680 (closed), #220758, but haven't found one for a blocking-like keyword.
I think that needs:if_exists as described in #30680 (closed) could work, with the downside that deploy-stage jobs would need to know about non-build dependencies.
Maybe we could preserve composability by patching the jobs in another include inside the top level Auto-DevOps.gitlab-ci.yml (e.g. Auto-DevOps-DAG-needs.gitlab-ci.yml), or directly in the top level template.