Recursive includes in .gitlab-ci.yml

changed title from Include yml declaration - "china doll style" to Include yml declaration - "Russian doll style"

changed the description

I just ran into the same issue. It seems to me that recursive (nested) includes doesn't work. Perhaps this is a better title? I was looking for it and stumbled in here.

Making a guess that this is a bug, but if the intention is not to support recursive includes, the documentation should probably state that it won't work. I found an early merge requests saying a (current, at that time) limitation was not supporting it. I'm happy to elaborate on my use case should motivation be required to support it.

I can definitely change the topic. I understand how this could possibly throw people off. But the use case for me is very simple unable to best yml files with include where the job is on the inner most yml file.

@sleath said: unable to best yml files with include where the job is on the inner most yml file.

Yeah, I face the same issue. I'm attempting to abstract away the stages declaration for all jobs and this fails pretty fast:

a.yml

stages:
 - foo
 - bar

b.yml

include: a.yml

myjob:
 - stage: foo

c.yml

include: b.yml

Above will fail and complain that stage foo is missing.

Edit: I "verified" (in lack of a better approach) that the nested require didn't fetch properly by changing c.yml to:

include:
 - a.yml
 - b.yml

..which works

Had a quick look at this. Hot path here: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/ci/external/processor.rb

It looks like it only merges the content from the included file and doesn't actually respect "include" statements in included files. ping @mayra-cabrera!

I think you would agree, if this were to be rolled out it would solve a bunch of use cases and it couldn't be a difficult revision.

thanks @sleath & @jbergstroem. This is an interesting idea!

...but if the intention is not to support recursive includes, the documentation should probably state that it won't work.

That's it's correct, we currently don't support recursive includes. The documentation should definitely mention that (I've included an update to the docs here https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5282)

Having said that, please upvote this issue so we can prioritize it for future milestones.

cc @bikebilly

changed title from Include yml declaration - "Russian doll style" to Include yml declaration - Recursive Includes

@mayra-cabrera In the yaml lint could you also include the proper error messages to reflect this not being supported or if there is a time table to support in future releases could you please include.

@sleath said: could you also include the proper error messages

I believe this is already covered elsewhere; saw a MR/Issue the other day (apologies for not searching it for you).

@mayra-cabrera said: The documentation should definitely mention that.

Thanks for that.

My short argument for why I think this should be classified as a bug (and as a result, perhaps documented as such): you define a keyword in a language (yaml) and users expect keywords to act consistently through their environment; much like you for instance can do object inheritance or even just define an array. By adding exceptions you make this slippery.

I understand that the implications of recursive merges are technically more challenging; be it recursive loops (include _self for instance), nesting limits and so on. It still seems like the "correct" thing to do; have a keyword in a language act consistently.

@mayra-cabrera I agree with @jbergstroem that

users expect keywords to act consistently through their environment,

I think that there should be at least a restraint around the amount of levels you can go. At the very least the ability should be 2 levels but definitely defined in docs. I can see how the includes could be expected behavior and I vote for this to be resolved in a future release.

In the yaml lint could you also include the proper error messages to reflect this not being supported or if there is a time table to support in future releases could you please include.

@sleath what do you think if we divide and conquer? As:

1 - We prepare this issue as a feature proposal for including recursivity in the include keyword

2 - We open another issue to also include the recursivity behavior in the Ci YAML Linter. As a follow up for the first point.

@mayra-cabrera said: what about giving control to the user?

My take would be to avoid exposing recurse control. I'm sure there's more than one disingenuous way of abusing this. How about just setting a hard limit of, say, 4 includes and expose a setting via gitlab.rb to increase this (to a sane degree).

Also: after arguing this as being a bug I'm starting to feel like it's indeed feature enhancement. It's a tricky problem and you took the sane approach.

@mayra-cabrera agreed do you want me to create the issue and feature proposal in both? Or were you planning on moving this there.

@sleath Please create another issue, following the "Feature proposal" template, for the Ci YAML Linter, and also update the description of this one :)

Feature proposal - https://gitlab.com/gitlab-org/gitlab-ee/issues/5640

thanks @sleath! It seems https://gitlab.com/gitlab-org/gitlab-ee/issues/5640 has the same description of this issue, so could you please close this one?

Recursive includes in .gitlab-ci.yml

Description

Designs

Child items 0

Activity

a.yml

b.yml

c.yml