Docker Registry: filesystem layer verification failed for digest sha256:xxx
Summary
(Since the last update,) docker can't pull images from bundled registry anymore
Steps to reproduce
GitLab CI tasks (running on multi-runner with docker+machine executor) that are using images from GitLab Docker Registry worked before and aren't working anymore
Actual behavior / Relevant logs and/or screenshots
What i've tried to fix that issue:
- Re-upload image -> error still appears
- Restarted docker-machines of autoscale runner (what clear all caches on the client side) -> error still appears
- Disabled docker registry-mirror -> error still appears
- Restarted GitLab server -> error still appears
Results of GitLab application Check
gitlab-rake gitlab:check SANITIZE=true
I, [2016-12-07T00:19:01.877533 #2960] INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors
Checking GitLab Shell ...
GitLab Shell version >= 4.0.0 ? ... OK (4.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
5/1 ... ok
5/2 ... ok
5/3 ... ok
2/4 ... ok
2/5 ... ok
33/6 ... ok
2/8 ... ok
34/9 ... ok
2/31 ... ok
4/33 ... ok
5/34 ... ok
5/35 ... ok
36/37 ... ok
8/38 ... ok
4/39 ... ok
4/40 ... ok
3/42 ... ok
1/43 ... ok
3/46 ... ok
3/47 ... ok
3/48 ... repository is empty
3/50 ... ok
15/51 ... ok
8/52 ... ok
3/53 ... ok
4/55 ... ok
36/56 ... ok
17/59 ... ok
17/60 ... ok
18/61 ... ok
18/63 ... ok
19/65 ... ok
19/66 ... ok
19/67 ... ok
19/68 ... ok
19/70 ... ok
19/71 ... ok
8/72 ... ok
19/76 ... ok
19/77 ... ok
3/78 ... ok
16/81 ... ok
3/82 ... ok
3/83 ... ok
5/84 ... ok
3/85 ... ok
3/86 ... ok
3/87 ... ok
19/88 ... ok
18/89 ... ok
3/90 ... ok
16/91 ... ok
36/92 ... ok
22/93 ... ok
22/94 ... ok
36/95 ... ok
22/96 ... ok
34/97 ... ok
19/99 ... ok
34/100 ... ok
36/102 ... ok
19/105 ... ok
22/106 ... ok
33/107 ... ok
22/108 ... ok
21/109 ... ok
56/110 ... ok
29/111 ... ok
22/112 ... ok
22/113 ... ok
36/114 ... ok
22/115 ... ok
22/116 ... ok
2/120 ... ok
30/121 ... ok
30/122 ... ok
32/123 ... ok
2/124 ... ok
34/126 ... ok
34/127 ... ok
34/128 ... ok
35/130 ... ok
35/131 ... ok
2/132 ... ok
36/133 ... ok
36/134 ... ok
36/135 ... ok
36/136 ... ok
36/137 ... ok
2/138 ... ok
2/139 ... ok
2/140 ... ok
2/141 ... ok
2/142 ... ok
2/143 ... ok
2/144 ... ok
2/145 ... ok
19/146 ... ok
38/147 ... ok
29/149 ... ok
2/150 ... ok
2/151 ... ok
2/152 ... ok
22/154 ... ok
22/156 ... ok
22/157 ... ok
2/158 ... ok
33/160 ... ok
23/161 ... ok
23/162 ... ok
19/163 ... ok
19/164 ... ok
36/165 ... ok
22/166 ... ok
19/167 ... ok
22/168 ... ok
36/169 ... ok
47/171 ... ok
32/172 ... ok
19/174 ... ok
2/175 ... ok
56/177 ... ok
48/178 ... ok
48/179 ... ok
2/180 ... ok
29/181 ... ok
50/182 ... ok
29/183 ... ok
23/185 ... ok
48/186 ... ok
4/188 ... ok
36/189 ... ok
4/190 ... ok
26/192 ... ok
38/193 ... ok
50/194 ... ok
2/195 ... ok
41/197 ... ok
3/198 ... ok
8/199 ... ok
22/200 ... ok
56/202 ... ok
36/203 ... ok
22/204 ... repository is empty
56/205 ... ok
61/206 ... ok
61/207 ... ok
19/208 ... ok
62/209 ... ok
62/210 ... ok
62/211 ... ok
66/212 ... ok
36/214 ... ok
69/215 ... ok
69/216 ... ok
59/217 ... ok
65/218 ... ok
73/219 ... ok
2/220 ... ok
2/221 ... ok
65/222 ... ok
65/223 ... ok
9/224 ... ok
36/227 ... ok
65/229 ... repository is empty
65/230 ... ok
8/231 ... ok
19/232 ... ok
19/233 ... ok
19/234 ... ok
38/235 ... ok
38/236 ... ok
38/237 ... ok
38/238 ... ok
38/239 ... ok
38/240 ... ok
38/241 ... ok
8/242 ... ok
19/243 ... ok
79/244 ... ok
22/245 ... ok
29/246 ... ok
29/247 ... ok
62/248 ... ok
4/249 ... ok
5/250 ... ok
5/251 ... ok
16/252 ... ok
16/253 ... ok
35/254 ... ok
62/257 ... ok
86/258 ... ok
86/259 ... ok
73/260 ... ok
73/261 ... ok
62/262 ... ok
85/263 ... ok
29/264 ... ok
80/265 ... repository is empty
33/266 ... ok
54/267 ... ok
33/268 ... ok
33/269 ... ok
9/270 ... ok
82/271 ... ok
85/272 ... ok
54/273 ... ok
54/275 ... ok
61/276 ... ok
61/277 ... ok
29/278 ... ok
34/279 ... ok
66/280 ... ok
34/281 ... ok
96/282 ... ok
98/283 ... ok
62/285 ... ok
62/287 ... ok
29/288 ... ok
62/289 ... ok
82/290 ... ok
62/292 ... ok
29/293 ... ok
62/294 ... ok
62/295 ... repository is empty
62/296 ... repository is empty
36/297 ... ok
19/298 ... ok
82/299 ... ok
62/300 ... ok
62/301 ... ok
54/302 ... ok
2/303 ... ok
109/304 ... ok
62/305 ... ok
109/306 ... ok
62/307 ... ok
29/308 ... ok
79/309 ... ok
62/310 ... ok
19/311 ... ok
62/313 ... ok
19/314 ... ok
111/315 ... ok
62/316 ... ok
19/317 ... ok
33/318 ... ok
36/319 ... ok
19/320 ... ok
36/321 ... ok
114/323 ... ok
115/324 ... ok
114/325 ... ok
36/326 ... ok
29/327 ... ok
109/328 ... ok
8/330 ... ok
8/333 ... ok
8/334 ... ok
33/335 ... ok
62/336 ... ok
54/337 ... repository is empty
33/338 ... ok
118/339 ... ok
118/340 ... ok
99/341 ... ok
8/342 ... ok
136/343 ... ok
29/344 ... ok
120/345 ... ok
29/346 ... ok
33/348 ... ok
84/349 ... ok
99/350 ... ok
99/351 ... repository is empty
99/352 ... ok
54/353 ... ok
62/354 ... ok
121/355 ... ok
62/356 ... ok
62/357 ... repository is empty
114/359 ... ok
114/360 ... repository is empty
73/361 ... ok
123/362 ... ok
136/363 ... ok
62/364 ... repository is empty
125/365 ... ok
125/366 ... ok
66/367 ... ok
126/368 ... ok
86/370 ... ok
86/371 ... ok
62/372 ... ok
123/374 ... ok
19/376 ... ok
108/377 ... ok
54/378 ... ok
62/379 ... ok
75/380 ... ok
29/381 ... ok
19/382 ... ok
33/383 ... ok
33/384 ... ok
73/385 ... ok
73/386 ... ok
126/387 ... ok
136/388 ... ok
82/389 ... ok
62/390 ... repository is empty
36/391 ... ok
136/393 ... ok
136/395 ... ok
82/396 ... ok
115/397 ... ok
62/399 ... ok
136/401 ... ok
62/402 ... ok
137/403 ... ok
62/404 ... ok
126/405 ... ok
126/406 ... ok
126/407 ... ok
134/408 ... ok
121/409 ... ok
84/411 ... ok
62/413 ... ok
134/415 ... ok
62/416 ... ok
126/417 ... ok
126/418 ... ok
19/420 ... ok
79/421 ... ok
126/422 ... ok
123/423 ... ok
19/424 ... ok
112/425 ... ok
146/426 ... ok
148/427 ... ok
148/428 ... repository is empty
29/429 ... ok
151/430 ... ok
151/431 ... ok
8/432 ... ok
1/433 ... ok
29/434 ... ok
8/435 ... ok
58/436 ... repository is empty
1/437 ... ok
22/438 ... ok
151/439 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped (omnibus-gitlab has no init script)
MailRoom running? ... can't check because of previous errors
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
5/1 ... yes
5/2 ... yes
5/3 ... yes
2/4 ... yes
2/5 ... yes
33/6 ... yes
2/8 ... yes
34/9 ... yes
2/31 ... yes
4/33 ... yes
5/34 ... yes
5/35 ... yes
36/37 ... yes
8/38 ... yes
4/39 ... yes
4/40 ... yes
3/42 ... yes
1/43 ... yes
3/46 ... yes
3/47 ... yes
3/48 ... yes
3/50 ... yes
15/51 ... yes
8/52 ... yes
3/53 ... yes
4/55 ... yes
36/56 ... yes
17/59 ... yes
17/60 ... yes
18/61 ... yes
18/63 ... yes
19/65 ... yes
19/66 ... yes
19/67 ... yes
19/68 ... yes
19/70 ... yes
19/71 ... yes
8/72 ... yes
19/76 ... yes
19/77 ... yes
3/78 ... yes
16/81 ... yes
3/82 ... yes
3/83 ... yes
5/84 ... yes
3/85 ... yes
3/86 ... yes
3/87 ... yes
19/88 ... yes
18/89 ... yes
3/90 ... yes
16/91 ... yes
36/92 ... yes
22/93 ... yes
22/94 ... yes
36/95 ... yes
22/96 ... yes
34/97 ... yes
19/99 ... yes
34/100 ... yes
36/102 ... yes
19/105 ... yes
22/106 ... yes
33/107 ... yes
22/108 ... yes
21/109 ... yes
56/110 ... yes
29/111 ... yes
22/112 ... yes
22/113 ... yes
36/114 ... yes
22/115 ... yes
22/116 ... yes
2/120 ... yes
30/121 ... yes
30/122 ... yes
32/123 ... yes
2/124 ... yes
34/126 ... yes
34/127 ... yes
34/128 ... yes
35/130 ... yes
35/131 ... yes
2/132 ... yes
36/133 ... yes
36/134 ... yes
36/135 ... yes
36/136 ... yes
36/137 ... yes
2/138 ... yes
2/139 ... yes
2/140 ... yes
2/141 ... yes
2/142 ... yes
2/143 ... yes
2/144 ... yes
2/145 ... yes
19/146 ... yes
38/147 ... yes
29/149 ... yes
2/150 ... yes
2/151 ... yes
2/152 ... yes
22/154 ... yes
22/156 ... yes
22/157 ... yes
2/158 ... yes
33/160 ... yes
23/161 ... yes
23/162 ... yes
19/163 ... yes
19/164 ... yes
36/165 ... yes
22/166 ... yes
19/167 ... yes
22/168 ... yes
36/169 ... yes
47/171 ... yes
32/172 ... yes
19/174 ... yes
2/175 ... yes
56/177 ... yes
48/178 ... yes
48/179 ... yes
2/180 ... yes
29/181 ... yes
50/182 ... yes
29/183 ... yes
23/185 ... yes
48/186 ... yes
4/188 ... yes
36/189 ... yes
4/190 ... yes
26/192 ... yes
38/193 ... yes
50/194 ... yes
2/195 ... yes
41/197 ... yes
3/198 ... yes
8/199 ... yes
22/200 ... yes
56/202 ... yes
36/203 ... yes
22/204 ... yes
56/205 ... yes
61/206 ... yes
61/207 ... yes
19/208 ... yes
62/209 ... yes
62/210 ... yes
62/211 ... yes
66/212 ... yes
36/214 ... yes
69/215 ... yes
69/216 ... yes
59/217 ... yes
65/218 ... yes
73/219 ... yes
2/220 ... yes
2/221 ... yes
65/222 ... yes
65/223 ... yes
9/224 ... yes
36/227 ... yes
65/229 ... yes
65/230 ... yes
8/231 ... yes
19/232 ... yes
19/233 ... yes
19/234 ... yes
38/235 ... yes
38/236 ... yes
38/237 ... yes
38/238 ... yes
38/239 ... yes
38/240 ... yes
38/241 ... yes
8/242 ... yes
19/243 ... yes
79/244 ... yes
22/245 ... yes
29/246 ... yes
29/247 ... yes
62/248 ... yes
4/249 ... yes
5/250 ... yes
5/251 ... yes
16/252 ... yes
16/253 ... yes
35/254 ... yes
62/257 ... yes
86/258 ... yes
86/259 ... yes
73/260 ... yes
73/261 ... yes
62/262 ... yes
85/263 ... yes
29/264 ... yes
80/265 ... yes
33/266 ... yes
54/267 ... yes
33/268 ... yes
33/269 ... yes
9/270 ... yes
82/271 ... yes
85/272 ... yes
54/273 ... yes
54/275 ... yes
61/276 ... yes
61/277 ... yes
29/278 ... yes
34/279 ... yes
66/280 ... yes
34/281 ... yes
96/282 ... yes
98/283 ... yes
62/285 ... yes
62/287 ... yes
29/288 ... yes
62/289 ... yes
82/290 ... yes
62/292 ... yes
29/293 ... yes
62/294 ... yes
62/295 ... yes
62/296 ... yes
36/297 ... yes
19/298 ... yes
82/299 ... yes
62/300 ... yes
62/301 ... yes
54/302 ... yes
2/303 ... yes
109/304 ... yes
62/305 ... yes
109/306 ... yes
62/307 ... yes
29/308 ... yes
79/309 ... yes
62/310 ... yes
19/311 ... yes
62/313 ... yes
19/314 ... yes
111/315 ... yes
62/316 ... yes
19/317 ... yes
33/318 ... yes
36/319 ... yes
19/320 ... yes
36/321 ... yes
114/323 ... yes
115/324 ... yes
114/325 ... yes
36/326 ... yes
29/327 ... yes
109/328 ... yes
8/330 ... yes
8/333 ... yes
8/334 ... yes
33/335 ... yes
62/336 ... yes
54/337 ... yes
33/338 ... yes
118/339 ... yes
118/340 ... yes
99/341 ... yes
8/342 ... yes
136/343 ... yes
29/344 ... yes
120/345 ... yes
29/346 ... yes
33/348 ... yes
84/349 ... yes
99/350 ... yes
99/351 ... yes
99/352 ... yes
54/353 ... yes
62/354 ... yes
121/355 ... yes
62/356 ... yes
62/357 ... yes
114/359 ... yes
114/360 ... yes
73/361 ... yes
123/362 ... yes
136/363 ... yes
62/364 ... yes
125/365 ... yes
125/366 ... yes
66/367 ... yes
126/368 ... yes
86/370 ... yes
86/371 ... yes
62/372 ... yes
123/374 ... yes
19/376 ... yes
108/377 ... yes
54/378 ... yes
62/379 ... yes
75/380 ... yes
29/381 ... yes
19/382 ... yes
33/383 ... yes
33/384 ... yes
73/385 ... yes
73/386 ... yes
126/387 ... yes
136/388 ... yes
82/389 ... yes
62/390 ... yes
36/391 ... yes
136/393 ... yes
136/395 ... yes
82/396 ... yes
115/397 ... yes
62/399 ... yes
136/401 ... yes
62/402 ... yes
137/403 ... yes
62/404 ... yes
126/405 ... yes
126/406 ... yes
126/407 ... yes
134/408 ... yes
121/409 ... yes
84/411 ... yes
62/413 ... yes
134/415 ... yes
62/416 ... yes
126/417 ... yes
126/418 ... yes
19/420 ... yes
79/421 ... yes
126/422 ... yes
123/423 ... yes
19/424 ... yes
112/425 ... yes
146/426 ... yes
148/427 ... yes
148/428 ... yes
29/429 ... yes
151/430 ... yes
151/431 ... yes
8/432 ... yes
1/433 ... yes
29/434 ... yes
8/435 ... yes
58/436 ... yes
1/437 ... yes
22/438 ... yes
151/439 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 47
Elasticsearch version 2.4.x? ... yes (2.4.2)
Elasticsearch has plugin delete-by-query installed? ... yes
Checking GitLab ... Finished
Results of GitLab environment info
I, [2016-12-07T00:20:33.872970 #3141] INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors
System information
System: Ubuntu 16.04
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.6.6
Bundler Version:1.13.6
Rake Version: 10.5.0
Sidekiq Version:4.2.1
GitLab information
Version: 8.14.3
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.2.18
URL: https://xxx
HTTP Clone URL: https://xxx/some-group/some-project.git
SSH Clone URL: git@xxx:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: github, gitlab, bitbucket, google_oauth2, facebook, twitter
GitLab Shell
Version: 4.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git