Skip to content

Track and check the number of jobs in the downstream FOSS pipeline

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Quoting !141897 (comment 1735958125)

Could we maybe have a follow-up issue to check the number of jobs in a few weeks, ideally with a dashboard (Sisense/Tableau) to check the number of jobs in the FOSS pipelines after we merged this MR? It would help us assess whether we need another iteration at it to reduce the number of jobs run. WDYT?

The main rational behind this is:

The more important thing I want to bring up is, this might mean that we will run way more FOSS jobs than before after this merge request. This scenario can happen:

Check the discussion thread for more details:

The following discussion from !141897 (merged) should be addressed:

  • @godfat-gitlab started a discussion: (+2 comments)

    This is very big and very difficult to maintain and review. The thing I've done here is looking through all the rules for all the as-if-foss jobs, and try to union them together. This process is a bit painful and I don't want to do it again, and I believe it's very difficult to review if this is correct, too.

    The fact is, we can't just blindly union them, because ordering does matter. We have quite a lot of when: never which makes it difficult to union.

    Anyway, I believe we can tweak as we go, even if this is not fully correct as for now.

    The more important thing I want to bring up is, this might mean that we will run way more FOSS jobs than before after this merge request. This scenario can happen:

    Previously:

    • We run rspec unit
    • We run build-assets-image
    • We run build-assets-image as-if-foss

    With this merge request, now it'll run:

    • We run rspec unit
    • We run build-assets-image
    • We run FOSS pipeline, with:
      • We run rspec unit
      • We run build-assets-image

    rspec unit is not just a job, it's a bunch of jobs, potentially including all other rspec jobs. So this can greatly increase the number of jobs we're running.

    Is this needed? Not sure. However this is easier to understand, because now we don't run "partial FOSS jobs". If we run FOSS, we run FOSS for all the jobs we're running. This makes it easier to predict what we run and the results will be more consistent, and the rules are easier to understand and easier to maintain.

    I do expect we might be running too much. In that case, perhaps we can tweak this big pile of rules. Iteration I think

Edited by 🤖 GitLab Bot 🤖