Backend: Rollout limits for directed acyclic graph on gitlab.com
<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=350398) </details> <!--IssueSummary end--> The new directed acyclic graph feature carries some risk of performance issues until https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31768 is implemented: > Note that one day one of the launch, we are temporarily limiting the > maximum number of jobs that a single job can need in the `needs:` array. Track > our [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7541) > for details on the current limit. This limits the following types of situations, where the number of a jobs that a single job wants to depend on is over the limit: ```yaml my_job: needs: [dependency1, dependency2, ...] ... ``` As such, we plan to roll it out as follows: - On the 22nd the feature will be shipped with a small limit (`5`). If someone goes over the limit, they will receive a pipeline creation error (`rspec: one job can only need 5 others, but you have listed 6. See needs keyword documentation for more details`) - @ayufan will monitor over a few days to determine the performance characteristics. - If the feature is performant, we will set the feature to a large limit (`50`). It will remain there for the next monitoring period. - If we see serious performance issues, we will disable the feature entirely. Users pipelines that use the DAG will still run, but will follow stage sequencing. The feature flag for toggling between `50` and `5` is `ci_dag_limit_needs`; it will limit to 5 when enabled. The feature flag for turning the feature on or off completely is `ci_dag_support`. We will need infrastructure support for the monitoring and toggling of necessary feature flags per the above plan. Once the limit has been removed, we should update the `needs:` section in the yaml docs to remove the reference to this issue.
issue