Project export fails

Summary

Exporting some projects fails.

Steps to reproduce

The projects are on a private server so I cannot give any link to the project to reproduce it.

Example Project

I cannot provide an example (as I cannot export the project).

What is the current bug behavior?

When I click on "Export project" in project settings / general I receive 4 emails with e.g. subject "CoreModel | Project export error" saying e.g.

Project CoreModel couldn't be exported. 
The errors we encountered were: 
•  No such file or directory @ rb_file_s_lstat - [FILTERED] 
— 

What is the expected correct behavior?

I should receive an email with the download link in.

Relevant logs and/or screenshots

In the folder /var/log/gitlab/gitlab-rails there is a sidekiq.log file. I have found errors such as:

{
  "severity": "WARN",
  "time": "2020-08-07T13:21:37.869Z",
  "error_class": "Gitlab::ImportExport::Error",
  "error_message": "No such file or directory @ rb_file_s_lstat - [FILTERED]",
  "context": "Job raised exception",
  "jobstr": "{\"class\":\"ProjectExportWorker\",\"args\":[9,176,null,{}],\"retry\":3,\"queue\":\"project_export\",\"backtrace\":5,\"status_expiration\":21600,\"jid\":\"7bcaa3a6640bdd1ab10877fb\",\"created_at\":1596806495.4087853,\"meta.user\":\"scp\",\"meta.project\":\"Security/CoreModel\",\"meta.root_namespace\":\"Security\",\"meta.caller_id\":\"ProjectsController#export\",\"correlation_id\":\"gYWiZ8B784a\",\"enqueued_at\":1596806495.4105225}",
  "class": "ProjectExportWorker",
  "args": [
    9,
    176,
    null,
    {}
  ],
  "retry": 3,
  "queue": "project_export",
  "backtrace": 5,
  "status_expiration": 21600,
  "jid": "7bcaa3a6640bdd1ab10877fb",
  "created_at": "2020-08-07T13:21:35.408Z",
  "meta.user": "scp",
  "meta.project": "Security/CoreModel",
  "meta.root_namespace": "Security",
  "meta.caller_id": "ProjectsController#export",
  "correlation_id": "gYWiZ8B784a",
  "enqueued_at": "2020-08-07T13:21:35.410Z",
  "error_backtrace": [
    "app/services/projects/import_export/export_service.rb:96:in `notify_error!'",
    "app/services/projects/import_export/export_service.rb:36:in `save_all!'",
    "app/services/projects/import_export/export_service.rb:13:in `execute'",
    "app/workers/project_export_worker.rb:19:in `perform'",
    "lib/gitlab/with_request_store.rb:7:in `with_request_store'",
    "lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"
  ]
}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.6.5p114
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.7
Git Version:    2.24.1
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        12.9.2
Revision:       ac5568eb5d8
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     10.12
URL:            https://iglab.it-innovation.soton.ac.uk
HTTP Clone URL: https://iglab.it-innovation.soton.ac.uk/some-group/some-project.git
SSH Clone URL:  git@iglab.it-innovation.soton.ac.uk:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        12.0.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.0.0 ? ... OK (12.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
4/1 ... yes
4/2 ... yes
4/3 ... yes
4/4 ... yes
4/5 ... yes
3/6 ... yes
3/7 ... yes
10/8 ... yes
10/9 ... yes
3/10 ... yes
6/11 ... yes
6/12 ... yes
6/13 ... yes
6/14 ... yes
6/15 ... yes
6/16 ... yes
6/17 ... yes
6/18 ... yes
8/19 ... yes
9/20 ... yes
10/21 ... yes
12/26 ... yes
11/28 ... yes
10/29 ... yes
4/30 ... yes
6/31 ... yes
4/32 ... yes
6/33 ... yes
8/34 ... yes
26/36 ... yes
26/41 ... yes
12/42 ... yes
29/43 ... yes
14/45 ... yes
3/47 ... yes
33/48 ... yes
8/49 ... yes
10/50 ... yes
12/52 ... yes
11/53 ... yes
14/54 ... yes
6/55 ... yes
39/56 ... yes
4/57 ... yes
14/58 ... yes
3/59 ... yes
3/60 ... yes
3/61 ... yes
10/62 ... yes
3/63 ... yes
40/64 ... yes
10/65 ... yes
10/67 ... yes
3/77 ... yes
20/78 ... yes
3/79 ... yes
3/80 ... yes
41/81 ... yes
43/82 ... yes
43/83 ... yes
43/85 ... yes
53/89 ... yes
43/90 ... yes
10/91 ... yes
10/92 ... yes
41/93 ... yes
41/94 ... yes
41/95 ... yes
46/96 ... yes
47/97 ... yes
10/98 ... yes
48/99 ... yes
43/101 ... yes
10/103 ... yes
50/105 ... yes
10/106 ... yes
43/107 ... yes
41/108 ... yes
41/109 ... yes
10/110 ... yes
50/111 ... yes
50/112 ... yes
43/113 ... yes
52/115 ... yes
20/116 ... yes
43/117 ... yes
41/118 ... yes
41/119 ... yes
41/120 ... yes
41/121 ... yes
43/123 ... yes
10/124 ... yes
21/125 ... yes
10/126 ... yes
41/127 ... yes
10/128 ... yes
10/129 ... yes
10/130 ... yes
50/132 ... yes
26/133 ... yes
14/134 ... yes
41/135 ... yes
41/136 ... yes
52/137 ... yes
53/138 ... yes
52/139 ... yes
54/142 ... yes
14/143 ... yes
20/144 ... yes
50/145 ... yes
10/147 ... yes
41/148 ... yes
16/149 ... yes
41/150 ... yes
50/152 ... yes
41/153 ... yes
61/156 ... yes
61/157 ... yes
61/162 ... yes
10/163 ... yes
14/164 ... yes
62/167 ... yes
61/168 ... yes
64/169 ... yes
115/171 ... yes
43/172 ... yes
65/173 ... yes
14/174 ... yes
41/175 ... yes
61/176 ... yes
115/177 ... yes
3/178 ... yes
41/179 ... yes
52/180 ... yes
66/181 ... yes
3/182 ... yes
41/183 ... yes
10/184 ... yes
68/187 ... yes
52/188 ... yes
61/189 ... yes
52/191 ... yes
3/192 ... yes
52/193 ... yes
115/194 ... yes
52/195 ... yes
52/197 ... yes
22/198 ... yes
21/199 ... yes
41/200 ... yes
41/201 ... yes
52/202 ... yes
64/203 ... yes
61/204 ... yes
45/205 ... yes
70/206 ... yes
70/207 ... yes
66/208 ... yes
67/209 ... yes
71/210 ... yes
64/211 ... yes
66/213 ... yes
66/214 ... yes
66/215 ... yes
70/216 ... yes
74/218 ... yes
115/220 ... yes
115/225 ... yes
66/226 ... yes
67/228 ... yes
14/231 ... yes
61/232 ... yes
66/234 ... yes
75/236 ... yes
61/237 ... yes
14/238 ... yes
43/239 ... yes
43/240 ... yes
66/245 ... yes
78/246 ... yes
78/247 ... yes
43/248 ... yes
67/249 ... yes
79/250 ... yes
79/251 ... yes
79/252 ... yes
79/253 ... yes
41/254 ... yes
79/255 ... yes
41/256 ... yes
75/257 ... yes
86/260 ... yes
41/261 ... yes
41/262 ... yes
86/263 ... yes
21/264 ... yes
41/266 ... yes
81/267 ... yes
75/271 ... yes
78/272 ... yes
41/273 ... yes
78/274 ... yes
78/275 ... yes
61/276 ... yes
81/277 ... yes
78/280 ... yes
66/281 ... yes
71/283 ... yes
61/284 ... yes
61/285 ... yes
41/286 ... yes
53/287 ... yes
78/288 ... yes
41/289 ... yes
52/290 ... yes
79/291 ... yes
68/292 ... yes
71/293 ... yes
71/294 ... yes
66/295 ... yes
14/297 ... yes
41/299 ... yes
78/300 ... yes
3/301 ... yes
79/302 ... yes
115/303 ... yes
26/304 ... yes
95/305 ... yes
96/306 ... yes
61/307 ... yes
64/308 ... yes
95/309 ... yes
98/313 ... yes
102/314 ... yes
101/315 ... yes
82/316 ... yes
102/317 ... yes
79/318 ... yes
104/319 ... yes
18/320 ... yes
61/321 ... yes
61/322 ... yes
96/323 ... yes
75/324 ... yes
75/325 ... yes
52/326 ... yes
96/327 ... yes
96/328 ... yes
75/329 ... yes
75/330 ... yes
103/331 ... yes
52/332 ... yes
75/333 ... yes
23/334 ... yes
110/336 ... yes
75/337 ... yes
115/338 ... yes
61/339 ... yes
61/340 ... yes
75/341 ... yes
75/342 ... yes
75/343 ... yes
75/344 ... yes
75/345 ... yes
96/346 ... yes
96/347 ... yes
113/348 ... yes
113/349 ... yes
115/350 ... yes
75/351 ... yes
77/352 ... yes
113/353 ... yes
113/354 ... yes
96/355 ... yes
103/358 ... yes
98/360 ... yes
98/362 ... yes
113/363 ... yes
75/364 ... yes
96/365 ... yes
75/366 ... yes
78/367 ... yes
75/368 ... yes
72/370 ... yes
113/371 ... yes
117/372 ... yes
117/373 ... yes
117/374 ... yes
72/375 ... yes
96/376 ... yes
75/377 ... yes
121/378 ... yes
120/379 ... yes
120/380 ... yes
120/381 ... yes
120/382 ... yes
72/383 ... yes
72/384 ... yes
26/385 ... yes
75/387 ... yes
75/388 ... yes
124/389 ... yes
124/390 ... yes
75/391 ... yes
120/392 ... yes
115/396 ... yes
75/397 ... yes
75/398 ... yes
118/399 ... yes
77/400 ... yes
61/401 ... yes
124/403 ... yes
61/404 ... yes
72/405 ... yes
115/406 ... yes
75/407 ... yes
77/408 ... yes
77/409 ... yes
77/410 ... yes
77/411 ... yes
127/413 ... yes
127/414 ... yes
130/415 ... yes
130/416 ... yes
130/417 ... yes
130/418 ... yes
130/419 ... yes
130/420 ... yes
130/421 ... yes
130/422 ... yes
130/423 ... yes
115/424 ... yes
79/425 ... yes
16/426 ... yes
61/427 ... yes
61/428 ... yes
96/429 ... yes
61/430 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.24.1)
Git user has default SSH configuration? ... yes
Active users: ... 27
Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

It's not clear why some projects successfully export and others do not. It might be to do with whether the project uses LFS (as I have not had a failure on projects that do not use LFS). I created a project and added a file using LFS and did successfully export it though so the LFS idea is not proven and just a hunch.