Job doesn't trigger if too many files changed

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

If there are too many changes in an MR, GitLab won't parse all of them and jobs won't be triggered off of files that weren't parsed.

Steps to reproduce

  1. Create a CI job that triggers if there are changes in a specific directory. Make this directory name start with a z to make the next step easier (so it is at the end of the 'to parse' list).
  2. Raise an MR against the project with more changes than is supported by GitLab (I've seen reference to this being >50MB?)
  3. View the MR in GitLab
  4. The overview notes that ####+ files were changed
  5. Only the jobs for changes that are parsed and displayed are run, any jobs for files that aren't parsed and displayed are NOT run.

Example Project

This is really related to an MR and I included a relevant CI job description below so I don't know that an example project here would be largely beneficial. If you disagree, I can look into making one.

What is the current bug behavior?

If there are too many changes in an MR, GitLab won't display all of them in the 'diff' view of an MR. This is understandable when just trying to view that diff (probably not that informative) but then any jobs that are supposed to trigger off of "hidden" files don't actually get triggered. It seems that these jobs are only being checked against this reduced set of changes, not the entire set of changes.

Related bug: #215720 (closed) This bug has the same problem but is looking for a way to view those hidden files. I don't necessarily need to view them (although that would be helpful) but I would expect the jobs to be triggered regardless.

My specific case is we have a 4 stage job. Stage 1 is a lint and always runs. Stage 2 is a base container build stage that builds only if there are changes to the base container (to save time in the more common case when there aren't changes here). Stage 3 uses either the already made base container or the new base container from stage 2 (if there were changes and a new one was made) and then compiles and runs tests inside this base container. Stage 4 builds the final image.

I went to make a release and merge a very large set of changes into master that included changes to the base container. But those changes never got parsed or displayed because there were too many other changes before them. Because of this, Stage 2 never triggered and Stage 3 was trying to run on the generally available base container when it needed to run in a new container with the new changes. I was able to work around this by first merging just the changes to the base container in so stage 2 would trigger and build a new generally available base container. I then re-ran the jobs for the large change and it passed (without stage 2) because all relevant changes were now already incorporated into the generally available base container and it no longer needed the new one to be built.

What is the expected correct behavior?

Ideally, I would like GitLab to explicitly check any files that are triggers in the CI definitions (e.g. in the only: changes sections) for changes as a standalone step instead of against the parsed/limited change set. This would check each job individually to see if it needs to trigger off of the ground truth, not a reduced set.

If this causes a performance issue or is difficult to implement, then I would like GitLab to check the triggers against the entire set of changes in the MR not just the parsed/limit section. Presumably this could just be done by checking that there are any changes in that path, not having to parse out all the changes in that path.

I absolutely appreciate the need to truncate and only display/parse up to a limit, but I would want jobs to be able to work somehow tangentially to the limit so they still trigger even if not all changes are parsed and displayed.

Relevant logs and/or screenshots

Very basic image that shows that not all files were parsed and are being displayed: image

Code snip-it for a CI job that should trigger on changes in a directory:

core:dev:
  image: ${REGISTRY}/${CORE_DEV_IMAGE}:${CI_COMMIT_REF_SLUG}
  only:
    refs:
      - merge_requests
    changes:
      - my/directory/**/*

In this case, with that many changes in the MR, this job won't be triggered.

Output of checks

This bug happened on GitLab.com. We don't host our own version, we run a private repo through you.

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

I don't have a good idea where this might be at all. I tried very briefly looking through https://gitlab.com/gitlab-org/gitlab but didn't see anything immediately. I would guess it's somewhere in here, but that is a pure guess: https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/services/ci

Edited by 🤖 GitLab Bot 🤖