Skip to content

Add support for extra submodule update flags

Nejc Habjan requested to merge siemens/gitlab-runner:feat/submodule-update-extra-flags into main

What does this MR do?

Adds a GIT_SUBMODULE_UPDATE_FLAGS for additional flags that can be appended to the git submodule update --init [--recursive] invocation when submodules are fetched, much like the existing GIT_FETCH_EXTRA_FLAGS.

Why was this MR needed?

Some people want to use the --remote flag (see #3011 (closed), although most of the discussion there misses how submodules really work). This goes a step further and enables adding any flag allowed with the update subcommand:
https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-update--init--remote-N--no-fetch--no-recommend-shallow-f--force--checkout--rebase--merge--referenceltrepositorygt--depthltdepthgt--recursive--jobsltngt--no-single-branch--ltpathgt82308203

For example --jobs for parallel fetches, --[no-]single-branch etc.

One thing I'd like to point out is that this would slightly overlap with the existing GIT_SUBMODULE_STRATEGY (for --recursive), GIT_DEPTH (when applied to submodules), and GIT_SUBMODULE_PATHS, because all of these could potentially be manually supplied to GIT_SUBMODULE_UPDATE_EXTRA_FLAGS as well. Because of this, I initially considered just having a GIT_SUBMODULE_UPDATE_STRATEGY, but I like the flexibility of this more, and it avoids future feature requests ("can I increase parallel jobs for submodules .. etc).

Are you okay with this overlap? I guess in some ways this could even replace some of the custom code for building the submodule update command, although this would be a breaking change. If needed I can try to provide some examples of how this might overlap.

If you decide this is the right approach I'll add a companion docs MR in the main repo (see https://gitlab.com/search?search=GIT_SUBMODULE_STRATEGY&group_id=9970&project_id=278964&scope=&search_code=true&snippets=false&repository_ref=master)

What's the best way to test this MR?

Add a runner built from this branch, register it and add the variable:

variables:
  GIT_SUBMODULE_STRATEGY: normal
  GIT_SUBMODULE_UPDATE_FLAGS: --remote --force --jobs 4

Enable CI_DEBUG_TRACE=true in CI variables and observe that these are passed to the update command.
I suppose the point here is not to also test git behavior but of ypu want to be sure, also enable GIT_TRACE=1 and observe that git runs 4 jobs (run-command.c:1628 run_processes_parallel: preparing to run up to 4 tasks).

Example: https://gitlab.com/nejch1/submodule-update-flags/-/jobs/1755771962#L606

++ echo 'Updating/initializing submodules with git depth set to 50...'
Updating/initializing submodules with git depth set to 50...
++ git submodule sync
++ git submodule foreach 'git clean -ffxd'
++ git submodule foreach 'git reset --hard'
++ git submodule update --init --depth 50 --remote --force --jobs 4
++ git submodule foreach 'git clean -ffxd'

What are the relevant issue numbers?

Closes #3011 (closed).
Closes #27277 (closed) (if people are happy supplying it as a flag - @steveazz would this be enough as you opened that issue?)

🛠 with ❤ at Siemens

Edited by Nejc Habjan

Merge request reports