Dynamically set parallelization values in Gemnasium pipelines

Problem

Currently, we manually set the parallelization values for the pipeline, which requires ongoing maintenance and adjustments to optimize performance and prevent job timeouts. Manually maintaining these values can be error-prone and may not adapt well to varying workloads and infrastructure changes.

Solution

To automate the setting of parallelization values, we propose dynamically calculating the parallel value in the parent job and passing it to the child pipeline. This approach would reduce maintenance overhead and adapt to changes in workloads more efficiently. Additionally, we should explore the feasibility of setting the parallel value to a CI variable. If this is not possible, we could consider generating the CI configuration of the child pipeline dynamically, although this may be complex and opaque.

Implementation Plan

To Be Confirmed (TBC)

  1. Investigate the feasibility of dynamically setting the parallel value using a CI variable.
  2. Explore the possibility of calculating the parallel value in the parent job and passing it to the child pipeline.
  3. Review the documentation on dynamic child pipelines and implement a proof of concept.
  4. If setting parallel via a CI variable is not possible, develop a strategy for dynamically generating the CI configuration of the child pipeline.
  5. Implement a notification mechanism to alert when @location.size differs from CI_NODE_TOTAL to aid in troubleshooting and adjustments.

References