Export project fails on large diff

Summary

Project export fails when project contains at least one merge request with at least one file diff > 64 KiB

Steps to reproduce

untested: Create a project, generate a change which produces a diff > 64 KiB (in my case it is 68 KiB) and try to export it.

Example Project

Creation of example project on gitlab.com seems not to be possible currently as that just results in a 503 (sic!)

What is the current bug behavior?

Export crashes.

What is the expected correct behavior?

Export should succeed.

Relevant logs and/or screenshots

https://forum.gitlab.com/t/issue-exporting-a-project-containing-a-diff-64-kib/9852

Gitlab::ImportExport::Error: (): found unexpected end of stream while scanning a quoted scalar at line 2 column 10

[
	"/usr/local/lib/ruby/2.3.0/psych.rb:379:in `parse'",
	"/usr/local/lib/ruby/2.3.0/psych.rb:379:in `parse_stream'",
	"/usr/local/lib/ruby/2.3.0/psych.rb:327:in `parse'",
	"/usr/local/lib/ruby/2.3.0/psych.rb:254:in `load'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/coders/yaml_column.rb:26:in `load'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/type/serialized.rb:19:in `type_cast_from_database'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:104:in `type_cast'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:42:in `original_value'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:37:in `value'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/transform_values.rb:11:in `block in transform_values'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/transform_values.rb:10:in `each'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/transform_values.rb:10:in `transform_values'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_set.rb:18:in `to_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_methods.rb:284:in `attributes'",
	"/home/git/gitlab/app/models/concerns/cache_markdown_field.rb:115:in `attributes'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:100:in `serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/serialization.rb:17:in `serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in `block (2 levels) in serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in `map'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in `block in serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:158:in `block in serializable_add_includes'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:156:in `each'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:156:in `serializable_add_includes'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:112:in `serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/serialization.rb:17:in `serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in `block (2 levels) in serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in `map'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:114:in `block in serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:158:in `block in serializable_add_includes'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:156:in `each'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:156:in `serializable_add_includes'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:112:in `serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/serialization.rb:17:in `serializable_hash'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activemodel-4.2.8/lib/active_model/serializers/json.rb:99:in `as_json'",
	"/home/git/gitlab/lib/gitlab/import_export/project_tree_saver.rb:34:in `project_json'",
	"/home/git/gitlab/lib/gitlab/import_export/project_tree_saver.rb:28:in `project_json_tree'",
	"/home/git/gitlab/lib/gitlab/import_export/project_tree_saver.rb:18:in `save'",
	"/home/git/gitlab/app/services/projects/import_export/export_service.rb:12:in `each'",
	"/home/git/gitlab/app/services/projects/import_export/export_service.rb:12:in `all?'",
	"/home/git/gitlab/app/services/projects/import_export/export_service.rb:12:in `save_all'",
	"/home/git/gitlab/app/services/projects/import_export/export_service.rb:6:in `execute'",
	"/home/git/gitlab/app/workers/project_export_worker.rb:11:in `perform'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:199:in `execute_job'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'",
	"/home/git/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'",
	"/home/git/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/server/active_record.rb:15:in `call'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.5.3/lib/raven/integrations/sidekiq.rb:7:in `call'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/middleware/chain.rb:133:in `invoke'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:169:in `block in process'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:141:in `block (6 levels) in dispatch'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/job_retry.rb:97:in `local'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:140:in `block (5 levels) in dispatch'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:136:in `block (4 levels) in dispatch'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:215:in `stats'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:131:in `block (3 levels) in dispatch'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/job_logger.rb:7:in `call'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:130:in `block (2 levels) in dispatch'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/job_retry.rb:72:in `global'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:129:in `block in dispatch'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/logging.rb:44:in `with_context'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/logging.rb:38:in `with_job_hash_context'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:128:in `dispatch'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:168:in `process'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:85:in `process_one'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/processor.rb:73:in `run'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:16:in `watchdog'",
	"/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/util.rb:25:in `block in safe_thread'"
]

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.5.2
Bundler Version:1.15.4
Rake Version:	12.0.0
Redis Version:	3.0.6
Git Version:	2.7.4
Sidekiq Version:5.0.4
Go Version:	go1.8.3 linux/amd64

GitLab information
Version:	9.5.2
Revision:	206fa16
Directory:	/home/git/gitlab
DB Adapter:	mysql2
URL:		http://gitlab.xxx.de
HTTP Clone URL:	http://gitlab.xxx.de/some-group/some-project.git
SSH Clone URL:	git@gitlab.xxx.de:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no


GitLab Shell
Version:	5.8.0
Repository storage paths:
  • default: /home/git/repositories Hooks: /home/git/gitlab-shell/hooks Git: /usr/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.5.2
Bundler Version:1.15.4
Rake Version:	12.0.0
Redis Version:	3.0.6
Git Version:	2.7.4
Sidekiq Version:5.0.4
Go Version:	go1.8.3 linux/amd64

GitLab information
Version:	9.5.2
Revision:	206fa16
Directory:	/home/git/gitlab
DB Adapter:	mysql2
URL:		http://gitlab.xxx.de
HTTP Clone URL:	http://gitlab.xxx.de/some-group/some-project.git
SSH Clone URL:	git@gitlab.xxx.de:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no


GitLab Shell
Version:	5.8.0
Repository storage paths:
  • default: /home/git/repositories Hooks: /home/git/gitlab-shell/hooks Git: /usr/bin/git root@gitlab:/home/git/gitlab# sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true Checking GitLab Shell ...

GitLab Shell version >= 5.8.0 ? ... OK (5.8.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 10/1 ... ok 10/3 ... ok 10/4 ... ok 10/6 ... ok 10/7 ... ok 10/8 ... ok 10/9 ... ok 10/11 ... ok 10/12 ... ok 10/13 ... ok 10/15 ... ok 10/17 ... ok 10/18 ... ok 10/20 ... ok 10/21 ... ok 10/22 ... ok 10/23 ... ok 5/24 ... repository is empty 10/25 ... ok 10/26 ... ok 10/27 ... ok 10/28 ... ok 10/29 ... ok 10/32 ... ok 9/33 ... ok 9/34 ... ok 10/35 ... ok 10/36 ... ok 10/37 ... ok 10/38 ... ok 10/39 ... ok 10/40 ... ok 10/41 ... ok 10/42 ... ok 10/43 ... ok 10/44 ... ok 10/45 ... ok 10/46 ... ok 10/47 ... ok 10/48 ... ok 16/49 ... ok 16/53 ... ok 16/54 ... ok 16/55 ... ok 16/56 ... ok 16/58 ... ok 17/59 ... ok 17/60 ... ok 17/61 ... ok 17/62 ... ok 16/63 ... ok 17/64 ... ok 10/65 ... ok 17/66 ... ok 17/67 ... ok 16/68 ... ok 2/69 ... ok 10/70 ... ok 17/71 ... ok 10/72 ... ok 10/73 ... ok 10/74 ... ok 17/75 ... ok 10/76 ... ok 10/77 ... repository is empty 10/78 ... ok 10/79 ... ok Running /home/git/gitlab-shell/bin/check Check GitLab API access: OK Access to /home/git/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... yes Init script up-to-date? ... yes Projects have namespace: ... 10/1 ... yes 10/3 ... yes 10/4 ... yes 10/6 ... yes 10/7 ... yes 10/8 ... yes 10/9 ... yes 10/11 ... yes 10/12 ... yes 10/13 ... yes 10/15 ... yes 10/17 ... yes 10/18 ... yes 10/20 ... yes 10/21 ... yes 10/22 ... yes 10/23 ... yes 5/24 ... yes 10/25 ... yes 10/26 ... yes 10/27 ... yes 10/28 ... yes 10/29 ... yes 10/32 ... yes 9/33 ... yes 9/34 ... yes 10/35 ... yes 10/36 ... yes 10/37 ... yes 10/38 ... yes 10/39 ... yes 10/40 ... yes 10/41 ... yes 10/42 ... yes 10/43 ... yes 10/44 ... yes 10/45 ... yes 10/46 ... yes 10/47 ... yes 10/48 ... yes 16/49 ... yes 16/53 ... yes 16/54 ... yes 16/55 ... yes 16/56 ... yes 16/58 ... yes 17/59 ... yes 17/60 ... yes 17/61 ... yes 17/62 ... yes 16/63 ... yes 17/64 ... yes 10/65 ... yes 17/66 ... yes 17/67 ... yes 16/68 ... yes 2/69 ... yes 10/70 ... yes 17/71 ... yes 10/72 ... yes 10/73 ... yes 10/74 ... yes 17/75 ... yes 10/76 ... yes 10/77 ... yes 10/78 ... yes 10/79 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.7.4) Active users: ... 5

Checking GitLab ... Finished

Edited by 🤖 GitLab Bot 🤖