Mirrored repositories missing their mirror settings, spamming logs
Summary
Most (not all) of our mirrored repositories in GitLab have lost their settings with relation to the upstream repo to mirror.
I also noticed that the logs for gitlab-rails/api_json.log we're getting hammered as if the mirroring was in a loop.
Steps to reproduce
Unsure, upgraded to GitLab 10.6 this morning and noticed it, but it could have happened in the recent past.
What is the current bug behaviour?
- Repositories that are mirrored are missing their mirror settings.
-
gitlab-rails/api_json.log
seems to suggest that repositories in this state are in some sort of loop. (see log example below for what its doing) - It looks like repositories are still being updated from their upstream mirror!?
Screenshot of a repository that was mirroring and now isn't:
What is the expected correct behaviour?
- Mirrored repositories should have the link to the mirror and mirror option ticked.
Screenshot after adding the upstream repository URL back in and enabling the mirror:
Relevant logs and/or screenshots
==> /var/log/gitlab/gitlab-rails/api_json.log <==
{"time":"2018-03-26T01:48:24.539Z","severity":"INFO","duration":42.64,"db":9.7,"view":32.94,"status":200,"method":"POST","path":"/api/v4/internal/allowed","params":{"action":"git-receive-pack","changes":"_any","gl_repository":null,"project":"mirrors/sysctl.git","protocol":"ssh","env":"{}","key_id":"[FILTERED]","secret_token":"[FILTERED]"},"host":"127.0.0.1","ip":"127.0.0.1","ua":"Ruby"}
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
I, [2018-03-26T12:48:24.542085 #104885] INFO -- : POST http://127.0.0.1:8080/api/v4/internal/allowed 0.05975
I, [2018-03-26T12:48:24.542538 #104885] INFO -- : gitlab-shell: executing git command <gitaly-receive-pack unix:/var/opt/gitlab/gitaly/gitaly.socket {"repository":{"storage_name":"default","relative_path":"mirrors/sysctl.git","git_object_directory":"","git_alternate_object_directories":[],"gl_repository":"project-194"},"gl_repository":"project-194","gl_id":"key-61","gl_username":"gitmirror"}> for user with key key-61.
==> /var/log/gitlab/gitaly/current <==
2018-03-26_01:48:24.56023 time="2018-03-26T01:48:24Z" level=info msg=spawn args="[/opt/gitlab/embedded/bin/git receive-pack /mnt/repositories/repositories/mirrors/sysctl.git]" grpc.method=SSHReceivePack grpc.request.repoPath=mirrors/sysctl.git grpc.request.repoStorage=default grpc.request.topLevelGroup=mirrors grpc.service=gitaly.SSHService path=/opt/gitlab/embedded/bin/git peer.address=@ pid=104894 span.kind=server system=grpc
2018-03-26_01:48:24.56908 time="2018-03-26T01:48:24Z" level=info msg="spawn complete" args="[/opt/gitlab/embedded/bin/git receive-pack /mnt/repositories/repositories/mirrors/sysctl.git]" command.exitCode=0 command.inblock=0 command.maxrss=31016 command.oublock=0 command.real_time_ms=9.706202000000001 command.system_time_ms=3.639 command.user_time_ms=0.912 grpc.method=SSHReceivePack grpc.request.repoPath=mirrors/sysctl.git grpc.request.repoStorage=default grpc.request.topLevelGroup=mirrors grpc.service=gitaly.SSHService path=/opt/gitlab/embedded/bin/git peer.address=@ span.kind=server system=grpc
2018-03-26_01:48:24.56917 time="2018-03-26T01:48:24Z" level=info msg="finished streaming call" grpc.code=OK grpc.method=SSHReceivePack grpc.request.repoPath=mirrors/sysctl.git grpc.request.repoStorage=default grpc.request.topLevelGroup=mirrors grpc.service=gitaly.SSHService grpc.time_ms=10 peer.address=@ span.kind=server system=grpc
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.6.0-ee Revision: ea9f9c5 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.8 URL: https://gitlab.redacted HTTP Clone URL: https://gitlab.redacted/some-group/some-project.git SSH Clone URL: git@gitlab.redacted:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 6.0.4 Repository storage paths:
- default: /mnt/repositories/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 6.0.4 ? ... OK (6.0.4) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 3/2 ... ok 4/3 ... ok 3/4 ... ok 3/5 ... ok 3/6 ... ok 3/7 ... ok 2/8 ... ok 3/9 ... ok 4/12 ... ok 4/13 ... ok 21/14 ... ok 4/16 ... ok 4/17 ... ok 21/18 ... ok 4/20 ... ok 4/24 ... ok 4/25 ... ok 4/27 ... ok 4/29 ... ok 4/30 ... ok 4/32 ... ok 4/34 ... ok 4/35 ... ok 4/36 ... ok 14/37 ... repository is empty 4/38 ... ok 4/40 ... ok 4/41 ... ok 4/42 ... ok 4/43 ... ok 4/44 ... ok 3/47 ... ok 4/48 ... ok 3/50 ... ok 3/51 ... ok 21/53 ... repository is empty 4/55 ... ok 4/56 ... ok 4/57 ... ok 36/59 ... repository is empty 35/60 ... ok 4/61 ... ok 38/64 ... repository is empty 4/65 ... ok 4/66 ... ok 4/68 ... ok 4/69 ... ok 3/71 ... ok 39/72 ... ok 35/74 ... ok 39/77 ... ok 4/78 ... ok 14/79 ... ok 4/81 ... ok 4/84 ... ok 4/85 ... ok 4/86 ... ok 46/90 ... ok 46/91 ... ok 4/92 ... ok 4/93 ... ok 3/96 ... ok 3/97 ... ok 14/98 ... ok 3/100 ... ok 3/104 ... ok 3/105 ... ok 3/106 ... ok 3/108 ... ok 4/109 ... ok 4/111 ... ok 4/112 ... ok 4/113 ... ok 4/114 ... ok 3/115 ... ok 39/116 ... ok 39/117 ... ok 48/118 ... ok 4/119 ... ok 39/120 ... ok 39/121 ... ok 4/122 ... ok 4/124 ... ok 4/131 ... ok 48/133 ... ok 4/134 ... ok 4/135 ... ok 21/136 ... ok 4/139 ... ok 21/140 ... ok 4/144 ... ok 21/145 ... repository is empty 4/146 ... ok 2/147 ... ok 3/149 ... ok 4/150 ... ok 21/151 ... ok 35/152 ... ok 14/153 ... ok 21/155 ... ok 7/156 ... ok 21/158 ... ok 78/159 ... repository is empty 78/161 ... repository is empty 4/163 ... ok 81/164 ... repository is empty 78/165 ... repository is empty 4/170 ... ok 4/171 ... ok 3/173 ... ok 4/178 ... ok 35/179 ... ok 35/181 ... ok 35/182 ... ok 35/183 ... ok 35/184 ... ok 35/185 ... ok 35/186 ... ok 35/187 ... ok 35/188 ... ok 35/189 ... ok 35/190 ... ok 35/191 ... ok 35/192 ... ok 35/193 ... ok 35/194 ... ok 35/195 ... ok 35/196 ... ok 35/197 ... ok 35/198 ... ok 35/199 ... ok 35/200 ... ok 5/201 ... ok 35/202 ... ok 35/203 ... ok 29/204 ... ok 35/205 ... ok 46/206 ... ok 35/207 ... ok 35/208 ... ok 3/210 ... ok 35/212 ... ok 29/215 ... ok 29/216 ... ok 35/217 ... ok 35/218 ... ok 4/219 ... ok 35/220 ... ok 4/221 ... ok 35/222 ... ok 35/223 ... ok 4/224 ... ok 3/225 ... ok 35/226 ... ok 35/227 ... ok 35/228 ... ok 35/229 ... ok 93/230 ... ok 35/233 ... ok 35/234 ... ok 35/235 ... ok 35/238 ... ok 35/241 ... ok 35/242 ... ok 35/243 ... ok 3/244 ... ok 35/245 ... ok 35/246 ... ok 14/247 ... ok 35/249 ... ok 64/250 ... ok 35/251 ... ok 4/253 ... ok 5/259 ... ok 5/260 ... ok 64/264 ... ok 35/274 ... ok 35/275 ... ok 4/281 ... ok 35/283 ... ok 35/285 ... ok 35/286 ... ok 35/288 ... ok 35/290 ... ok 3/291 ... ok 35/292 ... ok 35/293 ... ok 35/294 ... ok 3/295 ... ok 3/300 ... ok 3/301 ... ok 4/302 ... ok 35/304 ... ok 35/305 ... ok 35/306 ... ok 35/307 ... ok 103/308 ... ok 3/309 ... ok 35/310 ... ok 35/311 ... ok 5/312 ... ok 3/314 ... ok 14/317 ... ok 5/318 ... ok 3/319 ... ok 35/322 ... ok 103/323 ... ok 4/324 ... ok 7/325 ... ok 4/326 ... ok 35/327 ... ok 103/329 ... ok 3/331 ... repository is empty 103/332 ... repository is empty 35/339 ... ok 35/340 ... ok 35/342 ... ok 35/343 ... ok 35/352 ... ok 4/353 ... ok 4/355 ... ok 4/356 ... ok 16/358 ... ok 35/360 ... ok 3/361 ... ok 35/362 ... ok 35/363 ... ok 2/364 ... ok 103/365 ... ok 3/366 ... ok 35/367 ... ok 35/368 ... ok 35/369 ... ok 35/370 ... ok 35/371 ... ok 4/372 ... ok 35/373 ... ok 35/374 ... ok 108/375 ... ok 35/376 ... ok 35/377 ... ok 4/378 ... ok 103/379 ... ok 21/380 ... ok 35/381 ... ok 35/382 ... ok 5/383 ... ok 4/384 ... ok 3/385 ... ok 35/386 ... ok 35/387 ... ok 3/392 ... ok 35/393 ... ok 3/394 ... ok 35/395 ... ok 35/396 ... ok 35/397 ... ok 35/398 ... ok 35/399 ... ok 35/400 ... ok 35/401 ... ok 35/402 ... ok 35/403 ... ok 3/404 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results)
REDACTED
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/2 ... yes 4/3 ... yes 3/4 ... yes 3/5 ... yes 3/6 ... yes 3/7 ... yes 2/8 ... yes 3/9 ... yes 4/12 ... yes 4/13 ... yes 21/14 ... yes 4/16 ... yes 4/17 ... yes 21/18 ... yes 4/20 ... yes 4/24 ... yes 4/25 ... yes 4/27 ... yes 4/29 ... yes 4/30 ... yes 4/32 ... yes 4/34 ... yes 4/35 ... yes 4/36 ... yes 14/37 ... yes 4/38 ... yes 4/40 ... yes 4/41 ... yes 4/42 ... yes 4/43 ... yes 4/44 ... yes 3/47 ... yes 4/48 ... yes 3/50 ... yes 3/51 ... yes 21/53 ... yes 4/55 ... yes 4/56 ... yes 4/57 ... yes 36/59 ... yes 35/60 ... yes 4/61 ... yes 38/64 ... no Try fixing it: Migrate global projects For more information see: doc/update/5.4-to-6.0.md in section "#global-projects" Please fix the error above and rerun the checks. 4/65 ... yes 4/66 ... yes 4/68 ... yes 4/69 ... yes 3/71 ... yes 39/72 ... yes 35/74 ... yes 39/77 ... yes 4/78 ... yes 14/79 ... yes 4/81 ... yes 4/84 ... yes 4/85 ... yes 4/86 ... yes 46/90 ... yes 46/91 ... yes 4/92 ... yes 4/93 ... yes 3/96 ... yes 3/97 ... yes 14/98 ... yes 3/100 ... yes 3/104 ... yes 3/105 ... yes 3/106 ... yes 3/108 ... yes 4/109 ... yes 4/111 ... yes 4/112 ... yes 4/113 ... yes 4/114 ... yes 3/115 ... yes 39/116 ... yes 39/117 ... yes 48/118 ... yes 4/119 ... yes 39/120 ... yes 39/121 ... yes 4/122 ... yes 4/124 ... yes 4/131 ... yes 48/133 ... yes 4/134 ... yes 4/135 ... yes 21/136 ... yes 4/139 ... yes 21/140 ... yes 4/144 ... yes 21/145 ... yes 4/146 ... yes 2/147 ... yes 3/149 ... yes 4/150 ... yes 21/151 ... yes 35/152 ... yes 14/153 ... yes 21/155 ... yes 7/156 ... yes 21/158 ... yes 78/159 ... no Try fixing it: Migrate global projects For more information see: doc/update/5.4-to-6.0.md in section "#global-projects" Please fix the error above and rerun the checks. 78/161 ... no Try fixing it: Migrate global projects For more information see: doc/update/5.4-to-6.0.md in section "#global-projects" Please fix the error above and rerun the checks. 4/163 ... yes 81/164 ... no Try fixing it: Migrate global projects For more information see: doc/update/5.4-to-6.0.md in section "#global-projects" Please fix the error above and rerun the checks. 78/165 ... no Try fixing it: Migrate global projects For more information see: doc/update/5.4-to-6.0.md in section "#global-projects" Please fix the error above and rerun the checks. 4/170 ... yes 4/171 ... yes 3/173 ... yes 4/178 ... yes 35/179 ... yes 35/181 ... yes 35/182 ... yes 35/183 ... yes 35/184 ... yes 35/185 ... yes 35/186 ... yes 35/187 ... yes 35/188 ... yes 35/189 ... yes 35/190 ... yes 35/191 ... yes 35/192 ... yes 35/193 ... yes 35/194 ... yes 35/195 ... yes 35/196 ... yes 35/197 ... yes 35/198 ... yes 35/199 ... yes 35/200 ... yes 5/201 ... yes 35/202 ... yes 35/203 ... yes 29/204 ... yes 35/205 ... yes 46/206 ... yes 35/207 ... yes 35/208 ... yes 3/210 ... yes 35/212 ... yes 29/215 ... yes 29/216 ... yes 35/217 ... yes 35/218 ... yes 4/219 ... yes 35/220 ... yes 4/221 ... yes 35/222 ... yes 35/223 ... yes 4/224 ... yes 3/225 ... yes 35/226 ... yes 35/227 ... yes 35/228 ... yes 35/229 ... yes 93/230 ... yes 35/233 ... yes 35/234 ... yes 35/235 ... yes 35/238 ... yes 35/241 ... yes 35/242 ... yes 35/243 ... yes 3/244 ... yes 35/245 ... yes 35/246 ... yes 14/247 ... yes 35/249 ... yes 64/250 ... yes 35/251 ... yes 4/253 ... yes 5/259 ... yes 5/260 ... yes 64/264 ... yes 35/274 ... yes 35/275 ... yes 4/281 ... yes 35/283 ... yes 35/285 ... yes 35/286 ... yes 35/288 ... yes 35/290 ... yes 3/291 ... yes 35/292 ... yes 35/293 ... yes 35/294 ... yes 3/295 ... yes 3/300 ... yes 3/301 ... yes 4/302 ... yes 35/304 ... yes 35/305 ... yes 35/306 ... yes 35/307 ... yes 103/308 ... yes 3/309 ... yes 35/310 ... yes 35/311 ... yes 5/312 ... yes 3/314 ... yes 14/317 ... yes 5/318 ... yes 3/319 ... yes 35/322 ... yes 103/323 ... yes 4/324 ... yes 7/325 ... yes 4/326 ... yes 35/327 ... yes 103/329 ... yes 3/331 ... yes 103/332 ... yes 35/339 ... yes 35/340 ... yes 35/342 ... yes 35/343 ... yes 35/352 ... yes 4/353 ... yes 4/355 ... yes 4/356 ... yes 16/358 ... yes 35/360 ... yes 3/361 ... yes 35/362 ... yes 35/363 ... yes 2/364 ... yes 103/365 ... yes 3/366 ... yes 35/367 ... yes 35/368 ... yes 35/369 ... yes 35/370 ... yes 35/371 ... yes 4/372 ... yes 35/373 ... yes 35/374 ... yes 108/375 ... yes 35/376 ... yes 35/377 ... yes 4/378 ... yes 103/379 ... yes 21/380 ... yes 35/381 ... yes 35/382 ... yes 5/383 ... yes 4/384 ... yes 3/385 ... yes 35/386 ... yes 35/387 ... yes 3/392 ... yes 35/393 ... yes 3/394 ... yes 35/395 ... yes 35/396 ... yes 35/397 ... yes 35/398 ... yes 35/399 ... yes 35/400 ... yes 35/401 ... yes 35/402 ... yes 35/403 ... yes 3/404 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1522029952 sudo mv /var/opt/gitlab/.ssh/ask-izzy_key ~/gitlab-check-backup-1522029952 sudo mv /var/opt/gitlab/.ssh/config ~/gitlab-check-backup-1522029952 sudo mv /var/opt/gitlab/.ssh/authorized_keys_09_08_2017 ~/gitlab-check-backup-1522029952 sudo mv /var/opt/gitlab/.ssh/ask-izzy_key.pub ~/gitlab-check-backup-1522029952 sudo mv /var/opt/gitlab/.ssh/id_rsa.pub ~/gitlab-check-backup-1522029952 sudo mv /var/opt/gitlab/.ssh/authorized_keys_broken ~/gitlab-check-backup-1522029952 sudo mv /var/opt/gitlab/.ssh/id_rsa ~/gitlab-check-backup-1522029952 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 65 Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)
Checking GitLab ... Finished
~bug