Git submodules
Summary
Sorry in advance, I am not sure if this is a GitLab or GitLab runner issue.
I had problems with Git submodules using GitLab Runner 11.4.x and I had to use a workaround (gitlab-runner#3011 (closed)), which basically manually did the submodule initialisation and updating in a before_script
block. This worked fine until I upgraded to GitLab runner 11.5.2 where suddenly some projects broke and I got a weird error complaining about wrong submodule paths.
I though it was time to test the actual functionality and ditch the workaround, so I set
variables:
GIT_SUBMODULE_STRATEGY: recursive
however, this gave me the same error and it seems that it append ../../.git/...
to the submodule path, I don't understand why, like this (the exact same issue as mentioned in https://gitlab.com/gitlab-org/gitlab-ce/issues/34983):
fatal: Not a git repository: ../../.git/modules/the/submodule
fatal: Not a git repository: ../../.git/modules/the/submodule
Cannot chdir to , the toplevel of the working tree
and this:
Synchronizing submodule url for 'externals/aanet'
fatal: not a git repository: externals/aanet/../../.git/modules/externals/aanet
ERROR: Job failed: exit code 1
I can make it work when I set the Git strategy
to clone
instead of fetch
, but then the make
procedures are rebuilding everything all the time since the timestamps of the source files are modified and the cache is not taken into account (caching the actual source files would be a pretty bad idea).
So this now means we have to wait over an hour to get feedback from the CI for this project, since rebuilding everything takes that long.
Steps to reproduce
I tried to reproduce it with some example project, but was not able to. I also created a new branch of the affected project and set up the submodules from scratch and get the same error.
Example Project
Unfortunately I cannot make the project public.
What is the current bug behavior?
Git submodules are not initialised or updated.
What is the expected correct behavior?
Git submodules should be initialised and updated correctly.
Relevant logs and/or screenshots
This was the first error which appeared after upgrading from GitLab runner 11.4.1 (or something like that) to 11.5.1. Notice the ../../.git/modules/externals/aanet
which is appended to the path, the correct path of the submodule is just externals/aanet
:
Synchronizing submodule url for 'externals/aanet'
fatal: not a git repository: externals/aanet/../../.git/modules/externals/aanet
ERROR: Job failed: exit code 1
After that I reverted the old "hack" and used the intended configuration for submodules (with that env-var set to recursive
) and also deleted the runner caches, and got this error, printing the same wrong ../../.git/modules
paths:
Running with gitlab-runner 11.5.1 (7f00c780)
on ECAP Runner 4a8ecb4c
Using Docker executor with image docker.km3net.de/centos-full:7 ...
Pulling docker image docker.km3net.de/centos-full:7 ...
Using docker image sha256:d078d5d06f84816348e2c7fdd226e5146170e8939ece931f9b09ca9b84347e50 for docker.km3net.de/centos-full:7 ...
Running on runner-4a8ecb4c-project-223-concurrent-0 via pi1155...
Fetching changes...
HEAD is now at 0861bb57 Revert back workaround for git submodules
From https://git.km3net.de/common/jpp
0861bb57..6e9e20e2 2-add-gitlab-ci -> origin/2-add-gitlab-ci
Checking out 6e9e20e2 as 2-add-gitlab-ci...
Skipping Git submodules setup
Checking cache for 2-add-gitlab-ci-3...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ which git || (apk --no-cache add git)
/usr/bin/git
$ git submodule sync --recursive
Synchronizing submodule url for 'externals/aanet'
fatal: Not a git repository: ../../.git/modules/externals/aanet
fatal: Not a git repository: ../../.git/modules/externals/aanet
fatal: Not a git repository: ../../.git/modules/externals/aanet
fatal: Not a git repository: ../../.git/modules/externals/aanet
Cannot chdir to , the toplevel of the working tree
Synchronizing submodule url for 'externals/antares-dataformat'
fatal: Not a git repository: ../../.git/modules/externals/antares-dataformat
fatal: Not a git repository: ../../.git/modules/externals/antares-dataformat
fatal: Not a git repository: ../../.git/modules/externals/antares-dataformat
fatal: Not a git repository: ../../.git/modules/externals/antares-dataformat
Cannot chdir to , the toplevel of the working tree
Synchronizing submodule url for 'externals/chsm'
fatal: Not a git repository: ../../.git/modules/externals/chsm
fatal: Not a git repository: ../../.git/modules/externals/chsm
fatal: Not a git repository: ../../.git/modules/externals/chsm
fatal: Not a git repository: ../../.git/modules/externals/chsm
Cannot chdir to , the toplevel of the working tree
Synchronizing submodule url for 'externals/dbclient'
fatal: Not a git repository: ../../.git/modules/externals/dbclient
fatal: Not a git repository: ../../.git/modules/externals/dbclient
fatal: Not a git repository: ../../.git/modules/externals/dbclient
fatal: Not a git repository: ../../.git/modules/externals/dbclient
Cannot chdir to , the toplevel of the working tree
ERROR: Job failed: exit code 1
Then I re-reverted everything back and cleaned the caches etc. but it doesn't work anymore:
Running with gitlab-runner 11.5.1 (7f00c780)
on ECAP Runner 4a8ecb4c
Using Docker executor with image docker.km3net.de/centos-full:7 ...
Pulling docker image docker.km3net.de/centos-full:7 ...
Using docker image sha256:d078d5d06f84816348e2c7fdd226e5146170e8939ece931f9b09ca9b84347e50 for docker.km3net.de/centos-full:7 ...
Running on runner-4a8ecb4c-project-223-concurrent-0 via pi1155...
Fetching changes...
HEAD is now at fe42d09f Add docker tag to use the correct runner
From https://git.km3net.de/common/jpp
* [new branch] 5-git-transition -> origin/5-git-transition
Checking out 98da6e16 as 5-git-transition...
Updating/initializing submodules recursively...
Synchronizing submodule url for 'externals/aanet'
fatal: not a git repository: externals/aanet/../../.git/modules/externals/aanet
ERROR: Job failed: exit code 1
After setting the Git strategy to clone
, it magically works. Setting it back to fetch
, it fails again.
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.4.5p335 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.1 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.1 Go Version: unknown
GitLab information Version: 11.5.4-ee Revision: 62f48df Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.8 URL: https://git.km3net.de HTTP Clone URL: https://git.km3net.de/some-group/some-project.git SSH Clone URL: git@git.km3net.de:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2
GitLab Shell Version: 8.4.1 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 8.4.1 ? ... OK (8.4.1) hooks directories in repos are links: ... 4/1 ... ok 5/2 ... ok 6/3 ... ok 5/4 ... ok 5/5 ... ok 5/6 ... ok 5/7 ... ok 5/8 ... ok 5/9 ... ok 5/10 ... ok 5/11 ... ok 5/12 ... ok 5/13 ... ok 6/14 ... ok 7/15 ... repository is empty 6/16 ... ok 6/17 ... ok 6/18 ... ok 6/19 ... ok 6/20 ... ok 77/21 ... ok 3/22 ... ok 3/23 ... ok 4/24 ... ok 3/25 ... ok 3/26 ... ok 3/27 ... ok 3/28 ... ok 3/29 ... ok 2/30 ... ok 3/31 ... ok 2/32 ... ok 2/33 ... ok 2/34 ... ok 15/36 ... ok 15/37 ... ok 15/38 ... ok 15/39 ... ok 2/41 ... ok 17/46 ... ok 17/47 ... ok 17/48 ... ok 15/49 ... ok 19/50 ... ok 19/51 ... ok 20/52 ... ok 20/54 ... ok 20/55 ... ok 20/56 ... ok 21/58 ... repository is empty 15/59 ... ok 29/60 ... ok 34/61 ... ok 36/62 ... ok 22/63 ... ok 22/64 ... ok 32/66 ... ok 32/67 ... ok 41/68 ... ok 22/69 ... repository is empty 22/70 ... ok 2/73 ... ok 11/75 ... ok 22/76 ... ok 36/77 ... repository is empty 46/79 ... ok 46/80 ... ok 46/81 ... ok 46/82 ... ok 46/83 ... ok 46/84 ... ok 46/85 ... ok 41/86 ... ok 18/87 ... ok 18/88 ... ok 2/89 ... ok 43/90 ... ok 32/91 ... ok 6/92 ... ok 22/93 ... ok 6/95 ... ok 53/96 ... ok 6/98 ... ok 6/99 ... ok 53/100 ... ok 60/103 ... ok 22/104 ... ok 51/105 ... ok 6/106 ... ok 22/107 ... ok 3/108 ... ok 77/109 ... ok 56/114 ... ok 41/115 ... ok 65/116 ... ok 65/117 ... ok 65/118 ... ok 65/119 ... ok 65/120 ... ok 6/122 ... ok 35/123 ... ok 40/124 ... ok 6/125 ... ok 34/126 ... ok 6/127 ... ok 45/128 ... ok 67/129 ... ok 43/130 ... ok 22/131 ... ok 4/132 ... ok 69/133 ... ok 69/135 ... ok 46/136 ... ok 2/137 ... repository is empty 29/138 ... ok 69/139 ... ok 28/140 ... ok 14/141 ... ok 6/142 ... ok 2/143 ... ok 10/144 ... ok 10/145 ... ok 34/146 ... ok 60/148 ... ok 6/149 ... ok 76/152 ... ok 3/154 ... ok 2/155 ... ok 73/157 ... ok 4/158 ... ok 2/159 ... ok 32/160 ... ok 74/161 ... ok 32/162 ... ok 81/163 ... ok 48/164 ... ok 75/165 ... ok 75/166 ... ok 76/167 ... ok 80/168 ... ok 2/169 ... ok 2/176 ... ok 2/177 ... ok 32/178 ... ok 89/180 ... repository is empty 40/181 ... ok 75/182 ... ok 50/183 ... ok 75/184 ... ok 75/185 ... ok 75/186 ... ok 91/188 ... ok 92/189 ... ok 75/191 ... ok 22/192 ... ok 52/193 ... ok 34/194 ... ok 50/195 ... ok 2/196 ... ok 3/197 ... ok 2/198 ... ok 87/199 ... ok 46/200 ... ok 34/201 ... ok 88/203 ... ok 46/204 ... ok 2/205 ... ok 55/206 ... ok 46/207 ... ok 2/208 ... ok 51/209 ... ok 32/210 ... ok 34/211 ... ok 32/212 ... ok 22/214 ... ok 2/216 ... ok 98/217 ... ok 29/220 ... ok 65/221 ... ok 75/223 ... ok 75/224 ... ok 76/225 ... ok 76/227 ... ok 2/228 ... ok 76/234 ... ok 103/236 ... ok 75/237 ... ok 76/238 ... ok 76/239 ... ok 103/240 ... ok 2/241 ... ok 76/242 ... ok 103/243 ... ok 73/244 ... ok 103/246 ... ok 108/247 ... ok 104/248 ... ok 104/249 ... ok 104/250 ... ok 22/251 ... ok 103/253 ... ok 51/254 ... ok 2/255 ... ok 76/256 ... ok 2/257 ... ok 2/258 ... ok 46/259 ... ok 73/260 ... ok 73/261 ... ok 29/262 ... ok 29/263 ... ok 103/267 ... ok 103/268 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 5/2 ... yes 6/3 ... yes 5/4 ... yes 5/5 ... yes 5/6 ... yes 5/7 ... yes 5/8 ... yes 5/9 ... yes 5/10 ... yes 5/11 ... yes 5/12 ... yes 5/13 ... yes 6/14 ... yes 7/15 ... yes 6/16 ... yes 6/17 ... yes 6/18 ... yes 6/19 ... yes 6/20 ... yes 77/21 ... yes 3/22 ... yes 3/23 ... yes 4/24 ... yes 3/25 ... yes 3/26 ... yes 3/27 ... yes 3/28 ... yes 3/29 ... yes 2/30 ... yes 3/31 ... yes 2/32 ... yes 2/33 ... yes 2/34 ... yes 15/36 ... yes 15/37 ... yes 15/38 ... yes 15/39 ... yes 2/41 ... yes 17/46 ... yes 17/47 ... yes 17/48 ... yes 15/49 ... yes 19/50 ... yes 19/51 ... yes 20/52 ... yes 20/54 ... yes 20/55 ... yes 20/56 ... yes 21/58 ... yes 15/59 ... yes 29/60 ... yes 34/61 ... yes 36/62 ... yes 22/63 ... yes 22/64 ... yes 32/66 ... yes 32/67 ... yes 41/68 ... yes 22/69 ... yes 22/70 ... yes 2/73 ... yes 11/75 ... yes 22/76 ... yes 36/77 ... yes 46/79 ... yes 46/80 ... yes 46/81 ... yes 46/82 ... yes 46/83 ... yes 46/84 ... yes 46/85 ... yes 41/86 ... yes 18/87 ... yes 18/88 ... yes 2/89 ... yes 43/90 ... yes 32/91 ... yes 6/92 ... yes 22/93 ... yes 6/95 ... yes 53/96 ... yes 6/98 ... yes 6/99 ... yes 53/100 ... yes 60/103 ... yes 22/104 ... yes 51/105 ... yes 6/106 ... yes 22/107 ... yes 3/108 ... yes 77/109 ... yes 56/114 ... yes 41/115 ... yes 65/116 ... yes 65/117 ... yes 65/118 ... yes 65/119 ... yes 65/120 ... yes 6/122 ... yes 35/123 ... yes 40/124 ... yes 6/125 ... yes 34/126 ... yes 6/127 ... yes 45/128 ... yes 67/129 ... yes 43/130 ... yes 22/131 ... yes 4/132 ... yes 69/133 ... yes 69/135 ... yes 46/136 ... yes 2/137 ... yes 29/138 ... yes 69/139 ... yes 28/140 ... yes 14/141 ... yes 6/142 ... yes 2/143 ... yes 10/144 ... yes 10/145 ... yes 34/146 ... yes 60/148 ... yes 6/149 ... yes 76/152 ... yes 3/154 ... yes 2/155 ... yes 73/157 ... yes 4/158 ... yes 2/159 ... yes 32/160 ... yes 74/161 ... yes 32/162 ... yes 81/163 ... yes 48/164 ... yes 75/165 ... yes 75/166 ... yes 76/167 ... yes 80/168 ... yes 2/169 ... yes 2/176 ... yes 2/177 ... yes 32/178 ... yes 89/180 ... yes 40/181 ... yes 75/182 ... yes 50/183 ... yes 75/184 ... yes 75/185 ... yes 75/186 ... yes 91/188 ... yes 92/189 ... yes 75/191 ... yes 22/192 ... yes 52/193 ... yes 34/194 ... yes 50/195 ... yes 2/196 ... yes 3/197 ... yes 2/198 ... yes 87/199 ... yes 46/200 ... yes 34/201 ... yes 88/203 ... yes 46/204 ... yes 2/205 ... yes 55/206 ... yes 46/207 ... yes 2/208 ... yes 51/209 ... yes 32/210 ... yes 34/211 ... yes 32/212 ... yes 22/214 ... yes 2/216 ... yes 98/217 ... yes 29/220 ... yes 65/221 ... yes 75/223 ... yes 75/224 ... yes 76/225 ... yes 76/227 ... yes 2/228 ... yes 76/234 ... yes 103/236 ... yes 75/237 ... yes 76/238 ... yes 76/239 ... yes 103/240 ... yes 2/241 ... yes 76/242 ... yes 103/243 ... yes 73/244 ... yes 103/246 ... yes 108/247 ... yes 104/248 ... yes 104/249 ... yes 104/250 ... yes 22/251 ... yes 103/253 ... yes 51/254 ... yes 2/255 ... yes 76/256 ... yes 2/257 ... yes 2/258 ... yes 46/259 ... yes 73/260 ... yes 73/261 ... yes 29/262 ... yes 29/263 ... yes 103/267 ... yes 103/268 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.5) Git version >= 2.9.5 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 95 Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)
Checking GitLab ... Finished
Workarounds
1
This is just a workaround and not an acceptable fix since the timestamps of the source files are modified and the make system rebuilds everything all the time, but switching to git clone
as Git strategy works as expected.
2
Clear the cache containers the docker executor creates which forces the next job to clone the project.