New LFS content is not accessible in forks after upgrade to 13.5.1

After upgrade from 13.3 to 13.5.1 our GitLab instance we have been observing problems with accessing newly added files in lfs storage.

Summary

We have a fork flow. If anyone adds a new file in lfs in upstream, other developers can`t pull it from upstream. git pull upstream branch returns error - Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it. Since I can download this file from web UI and manually using lfs api it seems that something broken it smudge filter

However, if i try to disable smudge filter

git lfs install --skip-smudge
git pull upstream branch
git pull lfs upstream

lfs can download file

also, i can download it from commit

git lfs fetch upstream commit_id

Steps to reproduce

1. add a new file to lfs into upstream
2. try to pull it git pull upstream branch on another pc or git repo

What is the current bug behavior?

error occurs

 * branch                    2020.2     -> FETCH_HEAD
Downloading Bin/Framework4/ClassLibrary1.dll (5.6 KB)
Error downloading object: Bin/Framework4/ClassLibrary1.dll (a5793fe): Smudge error: Error downloading Bin/Framework4/ClassLibrary1.dll (a5793fe1a7b100bcdfcea91e28f55a02a05fd8408ca73a37adb5cc6c73cb0bb6): [a5793fe1a7b100bcdfcea91e28f55a02a05fd8408ca73a37adb5cc6c73cb0bb6] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it

What is the expected correct behavior?

the file should be downloaded by lfs correctly

Relevant logs and/or screenshots

lfs log 20201023T154430.1676737.log

Output of checks

We are using a self-hosted instance

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 18.04
Current User:   git
Using RVM:      no
Ruby Version:   2.6.6p146
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.9
Git Version:    2.28.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.5.1
Revision:       8ee0746f54c
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     11.7
URL:            http://gitserver
HTTP Clone URL: http://gitserver/some-group/some-project.git
SSH Clone URL:  git@gitserver:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.11.0
Repository storage paths:
- default:      /mnt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.11.0 ? ... OK (13.11.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/3 ... yes
138/17 ... yes
48/42 ... yes
83/80 ... yes
72/93 ... yes
88/103 ... yes
7/106 ... yes
126/107 ... yes
6/109 ... yes
6/110 ... yes
7/114 ... yes
7/116 ... yes
23/119 ... yes
23/120 ... yes
33/122 ... yes
103/123 ... yes
103/124 ... yes
7/128 ... yes
7/129 ... yes
17/130 ... yes
17/131 ... yes
17/132 ... yes
7/150 ... yes
254/151 ... yes
78/154 ... yes
14/156 ... yes
21/159 ... yes
7/163 ... yes
14/164 ... yes
95/165 ... yes
257/167 ... yes
167/172 ... yes
167/173 ... yes
81/179 ... yes
268/190 ... yes
268/209 ... yes
235/210 ... yes
136/214 ... yes
173/218 ... yes
72/230 ... yes
268/231 ... yes
188/236 ... yes
228/237 ... yes
252/238 ... yes
235/239 ... yes
234/241 ... yes
234/242 ... yes
173/245 ... yes
267/248 ... yes
136/250 ... yes
141/254 ... yes
177/255 ... yes
177/256 ... yes
222/258 ... yes
183/259 ... yes
183/260 ... yes
6/263 ... yes
252/264 ... yes
192/269 ... yes
192/270 ... yes
174/274 ... yes
269/279 ... yes
190/287 ... yes
267/291 ... yes
95/292 ... yes
148/295 ... yes
148/296 ... yes
269/317 ... yes
275/321 ... yes
72/325 ... yes
188/330 ... yes
275/331 ... yes
287/335 ... yes
291/344 ... yes
291/346 ... yes
2/365 ... yes
35/368 ... yes
141/377 ... yes
250/384 ... yes
89/387 ... yes
35/388 ... yes
250/392 ... yes
134/407 ... yes
174/408 ... yes
291/411 ... yes
136/412 ... yes
147/413 ... yes
228/416 ... yes
30/422 ... yes
81/423 ... yes
23/431 ... yes
307/436 ... yes
232/440 ... yes
215/449 ... yes
311/458 ... yes
311/464 ... yes
311/471 ... yes
215/472 ... yes
2/486 ... yes
313/497 ... yes
5/500 ... yes
2/502 ... yes
281/506 ... yes
172/507 ... yes
37/512 ... yes
2/513 ... yes
6/515 ... yes
8/522 ... yes
318/528 ... yes
2/529 ... yes
2/530 ... yes
323/531 ... yes
51/535 ... yes
83/547 ... yes
30/561 ... yes
89/565 ... yes
246/567 ... yes
54/575 ... yes
187/581 ... yes
305/586 ... yes
185/592 ... yes
246/593 ... yes
185/603 ... yes
2/606 ... yes
240/607 ... yes
54/611 ... yes
313/614 ... yes
2/615 ... yes
316/617 ... yes
281/622 ... yes
30/624 ... yes
181/632 ... yes
181/633 ... yes
321/634 ... yes
185/636 ... yes
351/640 ... yes
181/643 ... yes
232/645 ... yes
194/656 ... yes
7/663 ... yes
277/667 ... yes
51/669 ... yes
230/675 ... yes
156/679 ... yes
94/680 ... yes
19/681 ... yes
313/690 ... yes
35/691 ... yes
2/694 ... yes
35/699 ... yes
315/704 ... yes
152/708 ... yes
2/711 ... yes
64/713 ... yes
156/716 ... yes
356/717 ... yes
2/722 ... yes
48/727 ... yes
30/730 ... yes
83/735 ... yes
152/739 ... yes
179/741 ... yes
28/746 ... yes
250/751 ... yes
344/753 ... yes
371/756 ... yes
325/758 ... yes
353/760 ... yes
325/762 ... yes
156/766 ... yes
16/768 ... yes
14/769 ... yes
29/771 ... yes
21/773 ... yes
35/774 ... yes
13/775 ... yes
328/776 ... yes
344/777 ... yes
313/778 ... yes
156/779 ... yes
311/780 ... yes
250/782 ... yes
83/791 ... yes
156/793 ... yes
344/794 ... yes
338/799 ... yes
135/800 ... yes
50/804 ... yes
313/806 ... yes
30/807 ... yes
291/826 ... yes
45/828 ... yes
347/829 ... yes
321/831 ... yes
346/832 ... yes
374/833 ... yes
313/835 ... yes
244/837 ... yes
2/838 ... yes
307/839 ... yes
30/841 ... yes
67/842 ... yes
371/843 ... yes
50/844 ... yes
301/846 ... yes
50/847 ... yes
50/848 ... yes
370/850 ... yes
358/851 ... yes
244/855 ... yes
37/856 ... yes
315/857 ... yes
126/862 ... yes
342/863 ... yes
30/867 ... yes
2/869 ... yes
82/871 ... yes
55/875 ... yes
21/878 ... yes
55/888 ... yes
4/897 ... yes
334/899 ... yes
30/900 ... yes
313/906 ... yes
72/911 ... yes
264/914 ... yes
140/916 ... yes
246/918 ... yes
157/920 ... yes
70/921 ... yes
70/924 ... yes
110/925 ... yes
16/929 ... yes
185/933 ... yes
371/934 ... yes
250/935 ... yes
35/936 ... yes
306/939 ... yes
16/940 ... yes
127/941 ... yes
371/942 ... yes
16/958 ... yes
17/959 ... yes
196/960 ... yes
50/962 ... yes
39/963 ... yes
14/964 ... yes
157/965 ... yes
48/966 ... yes
103/967 ... yes
127/968 ... yes
33/969 ... yes
55/970 ... yes
126/971 ... yes
80/973 ... yes
57/974 ... yes
144/975 ... yes
35/976 ... yes
25/977 ... yes
18/978 ... yes
6/979 ... yes
5/980 ... yes
46/981 ... yes
302/983 ... yes
84/984 ... yes
170/985 ... yes
170/986 ... yes
170/987 ... yes
112/989 ... yes
154/991 ... yes
91/993 ... yes
135/994 ... yes
91/995 ... yes
8/996 ... yes
95/997 ... yes
8/998 ... yes
46/999 ... yes
85/1000 ... yes
59/1001 ... yes
117/1002 ... yes
57/1003 ... yes
180/1004 ... yes
117/1005 ... yes
180/1006 ... yes
180/1007 ... yes
368/1008 ... yes
393/1009 ... yes
363/1042 ... yes
369/1044 ... yes
277/1045 ... yes
82/1046 ... yes
9/1048 ... yes
334/1049 ... yes
334/1050 ... yes
4/1051 ... yes
232/1052 ... yes
201/1053 ... yes
232/1054 ... yes
364/1055 ... yes
364/1056 ... yes
47/1057 ... yes
107/1058 ... yes
47/1059 ... yes
107/1060 ... yes
104/1061 ... yes
107/1062 ... yes
249/1063 ... yes
34/1065 ... yes
104/1067 ... yes
344/1068 ... yes
64/1069 ... yes
147/1070 ... yes
110/1071 ... yes
322/1072 ... yes
262/1074 ... yes
85/1077 ... yes
227/1078 ... yes
150/1079 ... yes
21/1080 ... yes
19/1081 ... yes
120/1082 ... yes
59/1084 ... yes
36/1085 ... yes
195/1088 ... yes
385/1089 ... yes
142/1090 ... yes
101/1091 ... yes
72/1092 ... yes
44/1093 ... yes
116/1094 ... yes
366/1095 ... yes
155/1098 ... yes
155/1099 ... yes
81/1102 ... yes
168/1104 ... yes
184/1106 ... yes
184/1107 ... yes
184/1108 ... yes
382/1109 ... yes
207/1110 ... yes
24/1111 ... yes
24/1112 ... yes
24/1113 ... yes
26/1115 ... yes
90/1116 ... yes
109/1117 ... yes
90/1120 ... yes
120/1121 ... yes
379/1123 ... yes
175/1125 ... yes
336/1126 ... yes
336/1127 ... yes
269/1128 ... yes
37/1129 ... yes
265/1130 ... yes
328/1133 ... yes
328/1134 ... yes
140/1135 ... yes
32/1136 ... yes
59/1137 ... yes
216/1138 ... yes
216/1139 ... yes
216/1140 ... yes
362/1141 ... yes
13/1142 ... yes
28/1143 ... yes
53/1146 ... yes
53/1147 ... yes
70/1149 ... yes
128/1150 ... yes
70/1151 ... yes
70/1152 ... yes
70/1153 ... yes
128/1154 ... yes
128/1155 ... yes
212/1156 ... yes
16/1157 ... yes
111/1159 ... yes
64/1162 ... yes
395/1163 ... yes
45/1164 ... yes
45/1165 ... yes
137/1166 ... yes
137/1167 ... yes
367/1168 ... yes
137/1169 ... yes
34/1171 ... yes
155/1172 ... yes
153/1173 ... yes
194/1174 ... yes
153/1175 ... yes
243/1176 ... yes
243/1177 ... yes
11/1179 ... yes
54/1184 ... yes
59/1186 ... yes
336/1188 ... yes
194/1190 ... yes
215/1191 ... yes
240/1192 ... yes
172/1193 ... yes
111/1194 ... yes
99/1196 ... yes
199/1197 ... yes
26/1198 ... yes
156/1199 ... yes
37/1201 ... yes
240/1202 ... yes
56/1203 ... yes
56/1204 ... yes
369/1205 ... yes
131/1208 ... yes
70/1209 ... yes
66/1211 ... yes
358/1212 ... yes
238/1213 ... yes
394/1214 ... yes
244/1216 ... yes
147/1217 ... yes
390/1220 ... yes
264/1221 ... yes
391/1224 ... yes
101/1225 ... yes
2/1226 ... yes
264/1227 ... yes
322/1228 ... yes
185/1229 ... yes
215/1231 ... yes
215/1232 ... yes
101/1235 ... yes
267/1237 ... yes
16/1238 ... yes
260/1239 ... yes
9/1242 ... yes
134/1243 ... yes
389/1245 ... yes
57/1246 ... yes
185/1249 ... yes
287/1250 ... yes
172/1253 ... yes
222/1254 ... yes
232/1255 ... yes
89/1256 ... yes
179/1257 ... yes
89/1258 ... yes
195/1259 ... yes
81/1260 ... yes
344/1261 ... yes
313/1262 ... yes
317/1263 ... yes
398/1269 ... yes
222/1270 ... yes
172/1271 ... yes
398/1272 ... yes
79/1275 ... yes
402/1276 ... yes
13/1277 ... yes
402/1278 ... yes
2/1279 ... yes
179/1280 ... yes
2/1281 ... yes
35/1282 ... yes
55/1283 ... yes
170/1284 ... yes
82/1285 ... yes
344/1286 ... yes
17/1287 ... yes
134/1288 ... yes
81/1289 ... yes
53/1290 ... yes
28/1291 ... yes
405/1292 ... yes
405/1293 ... yes
153/1295 ... yes
153/1296 ... yes
153/1297 ... yes
344/1298 ... yes
102/1299 ... yes
90/1300 ... yes
358/1301 ... yes
363/1302 ... yes
153/1304 ... yes
243/1305 ... yes
181/1308 ... yes
406/1309 ... yes
45/1310 ... yes
405/1312 ... yes
394/1313 ... yes
407/1314 ... yes
406/1315 ... yes
128/1316 ... yes
128/1317 ... yes
102/1318 ... yes
401/1319 ... yes
218/1320 ... yes
333/1321 ... yes
203/1322 ... yes
2/1324 ... yes
410/1325 ... yes
410/1326 ... yes
155/1327 ... yes
155/1328 ... yes
155/1329 ... yes
155/1330 ... yes
140/1331 ... yes
412/1336 ... yes
46/1337 ... yes
35/1338 ... yes
205/1339 ... yes
242/1340 ... yes
371/1341 ... yes
2/1342 ... yes
404/1343 ... yes
181/1344 ... yes
2/1345 ... yes
382/1346 ... yes
407/1349 ... yes
185/1350 ... yes
364/1352 ... yes
37/1353 ... yes
37/1354 ... yes
360/1355 ... yes
344/1356 ... yes
45/1357 ... yes
215/1358 ... yes
322/1360 ... yes
95/1361 ... yes
36/1362 ... yes
30/1363 ... yes
194/1364 ... yes
179/1365 ... yes
240/1366 ... yes
21/1368 ... yes
179/1369 ... yes
156/1370 ... yes
344/1371 ... yes
344/1373 ... yes
194/1374 ... yes
371/1375 ... yes
185/1376 ... yes
189/1377 ... yes
370/1378 ... yes
371/1379 ... yes
194/1380 ... yes
158/1381 ... yes
407/1382 ... yes
344/1383 ... yes
240/1384 ... yes
419/1385 ... yes
45/1386 ... yes
385/1387 ... yes
45/1389 ... yes
297/1390 ... yes
305/1391 ... yes
45/1393 ... yes
55/1394 ... yes
382/1395 ... yes
407/1396 ... yes
2/1397 ... yes
59/1398 ... yes
250/1399 ... yes
2/1400 ... yes
181/1402 ... yes
98/1403 ... yes
311/1406 ... yes
156/1407 ... yes
363/1408 ... yes
30/1409 ... yes
52/1410 ... yes
257/1411 ... yes
82/1412 ... yes
35/1413 ... yes
427/1414 ... yes
180/1415 ... yes
167/1416 ... yes
55/1421 ... yes
107/1422 ... yes
80/1423 ... yes
104/1424 ... yes
87/1425 ... yes
75/1426 ... yes
35/1431 ... yes
311/1434 ... yes
187/1435 ... yes
134/1436 ... yes
194/1438 ... yes
37/1439 ... yes
57/1440 ... yes
277/1441 ... yes
277/1442 ... yes
362/1443 ... yes
181/1444 ... yes
37/1445 ... yes
432/1448 ... yes
432/1449 ... yes
111/1451 ... yes
175/1452 ... yes
14/1454 ... yes
287/1456 ... yes
89/1457 ... yes
89/1459 ... yes
187/1460 ... yes
240/1462 ... yes
427/1463 ... yes
427/1464 ... yes
434/1467 ... yes
434/1468 ... yes
289/1470 ... yes
175/1471 ... yes
168/1472 ... yes
34/1473 ... yes
368/1474 ... yes
369/1475 ... yes
363/1476 ... yes
24/1477 ... yes
216/1478 ... yes
126/1480 ... yes
297/1481 ... yes
16/1482 ... yes
45/1483 ... yes
275/1484 ... yes
194/1485 ... yes
194/1486 ... yes
26/1487 ... yes
427/1489 ... yes
86/1491 ... yes
382/1493 ... yes
267/1494 ... yes
437/1495 ... yes
242/1496 ... yes
158/1497 ... yes
418/1499 ... yes
137/1500 ... yes
389/1501 ... yes
157/1502 ... yes
379/1503 ... yes
438/1504 ... yes
59/1506 ... yes
439/1508 ... yes
362/1509 ... yes
20/1510 ... yes
307/1511 ... yes
315/1512 ... yes
440/1514 ... yes
410/1516 ... yes
194/1517 ... yes
130/1519 ... yes
427/1521 ... yes
148/1526 ... yes
391/1528 ... yes
391/1529 ... yes
445/1530 ... yes
410/1532 ... yes
45/1533 ... yes
407/1534 ... yes
242/1535 ... yes
448/1536 ... yes
449/1538 ... yes
35/1539 ... yes
130/1540 ... yes
50/1542 ... yes
21/1544 ... yes
45/1545 ... yes
185/1548 ... yes
240/1554 ... yes
344/1555 ... yes
48/1556 ... yes
215/1558 ... yes
344/1559 ... yes
14/1560 ... yes
362/1561 ... yes
112/1562 ... yes
30/1563 ... yes
11/1568 ... yes
344/1569 ... yes
48/1570 ... yes
2/1572 ... yes
30/1573 ... yes
50/1574 ... yes
16/1575 ... yes
32/1576 ... yes
35/1577 ... yes
127/1578 ... yes
137/1579 ... yes
368/1580 ... yes
175/1581 ... yes
26/1582 ... yes
216/1584 ... yes
242/1585 ... yes
297/1586 ... yes
34/1587 ... yes
123/1588 ... yes
412/1589 ... yes
18/1590 ... yes
187/1591 ... yes
287/1592 ... yes
289/1593 ... yes
89/1594 ... yes
21/1595 ... yes
98/1596 ... yes
177/1597 ... yes
267/1599 ... yes
442/1601 ... yes
156/1603 ... yes
442/1604 ... yes
59/1605 ... yes
5/1606 ... yes
61/1607 ... yes
409/1608 ... yes
61/1609 ... yes
315/1610 ... yes
344/1611 ... yes
435/1612 ... yes
382/1613 ... yes
369/1614 ... yes
168/1615 ... yes
179/1616 ... yes
363/1617 ... yes
185/1618 ... yes
37/1619 ... yes
37/1621 ... yes
441/1622 ... yes
16/1623 ... yes
86/1624 ... yes
30/1625 ... yes
215/1626 ... yes
8/1627 ... yes
330/1628 ... yes
446/1629 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 383
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes