GitLab.org / GitLab Community Edition

Added CI feature proposal labels

@antoine.gaillard13 Thanks for suggestion, we will consider it! /cc @markpundsack

Sounds like a great idea. One complication: the example given says "if a commit only modifies files in the test directory", but that's slightly different than running tests "if files in the test directory are changed". The latter is easy to do using the syntax above, but I don't think the former is.

Other common use cases would be to automatically ignore changes only to documentation or other files that don't affect the build/tests.

Milestone changed to Backlog

Changed title: [gitlab ci] Run stages/jobs only when there are modifications on a given path → Run jobs only when there are modifications on a given path

@markpundsack You're right my example is wrong. It should be something like

build:
  stage: build
  only:
    - paths:
      - "^src/.*"
test:
  stage: test
  only:
    - paths:
      - "^src/.*"
      - "^test/.*"

@antoine.gaillard13

I think that it may be a little tricky when we consider the incremental changes.

If we have a merge request that has 3 different commits, where:

1. modify tests,
2. modify documentation,
3. modify CHANGELOG,

And we have three seperate pushes for each of the commits and we are processing changes incrementally. How the flow of creating builds should look like:

1. Should we compare changes to previous push?
2. Should we compare against branch that we are merging to?

intuitively it seems that it should take into account the branch that we are merging to.

Yeah, I think comparing to the target branch, at the point of the branch, makes sense. Basically, whatever changes are included in a merge request.

Of course the real problem is that we currently run tests against branches, not merge requests, so we don't know what the target branch is.

mentioned in issue #19555

@ayufan Yes you're right. I'd also like to point out that we have to handle artifacts. If we have something like:

build:
  stage: build
  only:
    - paths:
      - "^src/.*"
  artifacts:
    - paths:
       - "bin/"
test:
 stage: test
 only:
   - paths:
     - "^src/.*"
     - "^test/.*"

and we run black box tests in the test stage against binaries built during the first stage. If we only modify test subdirectory, we should download artifacts from the last parent commit that generated them.

Having only: paths seems like a nice feature. So 👍

same as #19813 (closed)

mentioned in issue #19813 (closed)

mentioned in issue #18157

+1 This would make it possible for me to independently build/test/deploy the subprojects that live in my master repository.

+1

Is possible to store a hash of the content of the file/folder and compare later if it change?

@markpundsack this came up in slack today: https://gitlab.slack.com/archives/C027N716H/p1496775158700189 Wanted to get it back on your Radar <3.

Thanks @lbot. It's definitely still on our radar, but it's pretty complicated and not going to happen soon. But maybe we can open it up for Accepting Merge Requests.

Also, now that I'm rereading old comments, I wonder if we do want to test for content of each push, not compare vs target branch. The latter is especially hard because we don't know the target, so for now, the former is all we could do anyway. But also, I think it might be the right answer regardless. If I push a bunch of commits, and one more push contains just docs changes, there's no reason to re-run all builds and tests just because the branch contains other changes. Likewise, for master, there is no target branch, so paths would be useless unless they applied to individual commits in the push.

I do see the concern mentioned in #19232 (comment 13096248) though regarding artifacts. It might work to grab the latest artifact from the same ref, but I imagine that backfiring at some point. But that's probably semantically the same as what you're doing with Jenkins and persistent artifacts/caches.

added direction test labels

changed the description

added Accepting Merge Requests label

changed title from Run jobs only when there are modifications on a given path to Run jobs only/except when there are modifications on a given path

I cannot read what is in that slack link, because it is behind a login - can that content be shared here?

mentioned in issue #18041

To be honest I think this should have more priority. Companies are emerging towards more modular systems architecture with microservices and monorepos and more and more people are adopting GitLab CI for its CI/CD process. Using GitLab CI with a large monorepo today is really bad because you don't want to build everything whenever there are changes. You would want to trigger the pipeline/stage/job based on only/except some file path (glob pattern).

With #18041 you could also create a script to check changes.

added pipeline and removed test labels

changed milestone to Next 3-6 months

This would be great for our cloudformation templates

mentioned in issue #23455