Uploads with long filenames on MySQL fail
Summary
Uploads with long filenames on MySQL fail.
Steps to reproduce
On a GitLab instance >= v9.0, using MySQL, upload a file with a 255 character long filename to a Markdown field (issue, MR, comment).
An insert to the uploads table fails because uploads.path
is varchar(255)
for Rails string
fields by default, and the path is the filename plus some directories.
I think this also affects appearance logo and header logo uploads but not avatars since avatar filenames get renamed to 'avatar' by the frontend.
What is the current bug behavior?
Upload dies.
What is the expected correct behavior?
Upload happens.
I believe uploads.path
needs to handle at least 298 characters for the current Uploader schemes.
Max length | Path kind of looks like |
---|---|
297 | uploads/-/system/appearance/header_logo/1/255_char_filename |
297 | uploads/-/system/project/avatar/999999999/255_char_filename |
295 | uploads/-/system/group/avatar/999999999/255_char_filename |
272 | 16_character_hex/255_char_filename |
49 | uploads/-/system/user/avatar/999999999/avatar.png |
Relevant logs and/or screenshots
Locally on master branch on Nov 14, 2017
Started POST "/h5bp/html5-boilerplate/uploads" for 127.0.0.1 at 2017-11-14 19:50:51 -0800
Project Load (0.6ms) SELECT `projects`.* FROM `projects` INNER JOIN `routes` ON `routes`.`source_id` = `projects`.`id` AND `routes`.`source_type` = 'Project' WHERE ((routes.path = 'h5bp/html5-boilerplate')) ORDER BY (CASE WHEN BINARY routes.path = 'h5bp/html5-boilerplate' THEN 0 ELSE 1 END) LIMIT 1
CACHE (0.0ms) SELECT `projects`.* FROM `projects` INNER JOIN `routes` ON `routes`.`source_id` = `projects`.`id` AND `routes`.`source_type` = 'Project' WHERE ((routes.path = 'h5bp/html5-boilerplate')) ORDER BY (CASE WHEN BINARY routes.path = 'h5bp/html5-boilerplate' THEN 0 ELSE 1 END) LIMIT 1 [["source_type", "Project"]]
Processing by Projects::UploadsController#create as JSON
Parameters: {"file.path"=>"/Users/monozok/Developer/gitlab-development-kit/gitlab/public/uploads/tmp/multipart-012013196/255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename.jpg", "file.name"=>"255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename.jpg", "file"=>#<UploadedFile:0x00007fa2cf082c98 @content_type="application/octet-stream", @original_filename="255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename.jpg", @tempfile=#<File:/Users/monozok/Developer/gitlab-development-kit/gitlab/public/uploads/tmp/multipart-012013196/255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename.jpg>>, "namespace_id"=>"h5bp", "project_id"=>"html5-boilerplate"}
User Load (0.6ms) SELECT `users`.* FROM `users` WHERE `users`.`id` = 1 ORDER BY `users`.`id` ASC LIMIT 1
CACHE (0.0ms) SELECT `projects`.* FROM `projects` INNER JOIN `routes` ON `routes`.`source_id` = `projects`.`id` AND `routes`.`source_type` = 'Project' WHERE ((routes.path = 'h5bp/html5-boilerplate')) ORDER BY (CASE WHEN BINARY routes.path = 'h5bp/html5-boilerplate' THEN 0 ELSE 1 END) LIMIT 1 [["source_type", "Project"]]
Route Load (0.4ms) SELECT `routes`.* FROM `routes` WHERE `routes`.`source_id` = 16 AND `routes`.`source_type` = 'Project' LIMIT 1
Upload Load (0.5ms) SELECT `uploads`.* FROM `uploads` WHERE `uploads`.`path` = 'c60396973cf7ca24c4c207365190b7fb/255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename.jpg'
(0.2ms) BEGIN
SQL (1.5ms) INSERT INTO `uploads` (`size`, `path`, `model_id`, `model_type`, `uploader`, `checksum`, `created_at`) VALUES (335679, 'c60396973cf7ca24c4c207365190b7fb/255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename_255_character_long_filename.jpg', 16, 'Project', 'FileUploader', 'fdb2dfdef19b5d4add6ccd575dd13be59270cd1ed2c89f93658f282635f55d4a', '2017-11-15 03:50:52')
(0.2ms) ROLLBACK
Completed 500 Internal Server Error in 59ms (ActiveRecord: 4.6ms)
Mysql2::Error - Data too long for column 'path' at row 1:
mysql2 (0.4.5) lib/mysql2/client.rb:120:in `block in query'
mysql2 (0.4.5) lib/mysql2/client.rb:119:in `query'
peek-mysql2 (1.1.0) lib/peek/views/mysql2.rb:14:in `query_with_timing'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
activesupport (4.2.8) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'
activerecord (4.2.8) lib/active_record/connection_adapters/mysql2_adapter.rb:225:in `execute'
config/initializers/connection_fix.rb:20:in `execute'
activerecord (4.2.8) lib/active_record/connection_adapters/mysql2_adapter.rb:242:in `exec_insert'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract/database_statements.rb:108:in `insert'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `insert'
activerecord (4.2.8) lib/active_record/relation.rb:64:in `insert'
activerecord (4.2.8) lib/active_record/persistence.rb:524:in `_create_record'
activerecord (4.2.8) lib/active_record/counter_cache.rb:139:in `_create_record'
config/initializers/active_record_locking.rb:12:in `_create_record'
activerecord (4.2.8) lib/active_record/attribute_methods/dirty.rb:133:in `_create_record'
activerecord (4.2.8) lib/active_record/callbacks.rb:306:in `block in _create_record'
activesupport (4.2.8) lib/active_support/callbacks.rb:88:in `__run_callbacks__'
activesupport (4.2.8) lib/active_support/callbacks.rb:778:in `_run_create_callbacks'
activerecord (4.2.8) lib/active_record/callbacks.rb:306:in `_create_record'
activerecord (4.2.8) lib/active_record/timestamp.rb:57:in `_create_record'
activerecord (4.2.8) lib/active_record/persistence.rb:504:in `create_or_update'
activerecord (4.2.8) lib/active_record/callbacks.rb:302:in `block in create_or_update'
activesupport (4.2.8) lib/active_support/callbacks.rb:117:in `call'
activesupport (4.2.8) lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
activesupport (4.2.8) lib/active_support/callbacks.rb:505:in `call'
activesupport (4.2.8) lib/active_support/callbacks.rb:92:in `__run_callbacks__'
activesupport (4.2.8) lib/active_support/callbacks.rb:778:in `_run_save_callbacks'
activerecord (4.2.8) lib/active_record/callbacks.rb:302:in `create_or_update'
activerecord (4.2.8) lib/active_record/persistence.rb:120:in `save'
activerecord (4.2.8) lib/active_record/validations.rb:37:in `save'
activerecord (4.2.8) lib/active_record/attribute_methods/dirty.rb:21:in `save'
activerecord (4.2.8) lib/active_record/transactions.rb:286:in `block (2 levels) in save'
activerecord (4.2.8) lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
activerecord (4.2.8) lib/active_record/transactions.rb:220:in `transaction'
activerecord (4.2.8) lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
activerecord (4.2.8) lib/active_record/transactions.rb:286:in `block in save'
activerecord (4.2.8) lib/active_record/transactions.rb:301:in `rollback_active_record_state!'
activerecord (4.2.8) lib/active_record/transactions.rb:285:in `save'
activerecord (4.2.8) lib/active_record/persistence.rb:34:in `create'
app/models/upload.rb:22:in `record'
app/uploaders/records_uploads.rb:21:in `record_upload'
carrierwave (1.2.1) lib/carrierwave/uploader/callbacks.rb:16:in `block in with_callbacks'
carrierwave (1.2.1) lib/carrierwave/uploader/callbacks.rb:16:in `with_callbacks'
carrierwave (1.2.1) lib/carrierwave/uploader/store.rb:56:in `store!'
app/services/upload_service.rb:12:in `execute'
app/controllers/concerns/uploads_actions.rb:3:in `create'
actionpack (4.2.8) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (4.2.8) lib/abstract_controller/base.rb:198:in `process_action'
actionpack (4.2.8) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (4.2.8) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
activesupport (4.2.8) lib/active_support/callbacks.rb:117:in `call'
activesupport (4.2.8) lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
activesupport (4.2.8) lib/active_support/callbacks.rb:505:in `call'
activesupport (4.2.8) lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
activesupport (4.2.8) lib/active_support/callbacks.rb:313:in `block (2 levels) in halting'
lib/gitlab/i18n.rb:47:in `with_locale'
lib/gitlab/i18n.rb:53:in `with_user_locale'
app/controllers/application_controller.rb:336:in `set_locale'
activesupport (4.2.8) lib/active_support/callbacks.rb:432:in `block in make_lambda'
activesupport (4.2.8) lib/active_support/callbacks.rb:312:in `block in halting'
activesupport (4.2.8) lib/active_support/callbacks.rb:497:in `block in around'
activesupport (4.2.8) lib/active_support/callbacks.rb:505:in `call'
activesupport (4.2.8) lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
activesupport (4.2.8) lib/active_support/callbacks.rb:298:in `block in halting_and_conditional'
activesupport (4.2.8) lib/active_support/callbacks.rb:497:in `block in around'
activesupport (4.2.8) lib/active_support/callbacks.rb:505:in `call'
activesupport (4.2.8) lib/active_support/callbacks.rb:92:in `__run_callbacks__'
activesupport (4.2.8) lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
activesupport (4.2.8) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (4.2.8) lib/abstract_controller/callbacks.rb:19:in `process_action'
actionpack (4.2.8) lib/action_controller/metal/rescue.rb:29:in `process_action'
actionpack (4.2.8) lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
activesupport (4.2.8) lib/active_support/notifications.rb:164:in `block in instrument'
activesupport (4.2.8) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (4.2.8) lib/active_support/notifications.rb:164:in `instrument'
actionpack (4.2.8) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
actionpack (4.2.8) lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
activerecord (4.2.8) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (4.2.8) lib/abstract_controller/base.rb:137:in `process'
actionview (4.2.8) lib/action_view/rendering.rb:30:in `process'
actionpack (4.2.8) lib/action_controller/metal.rb:196:in `dispatch'
actionpack (4.2.8) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
actionpack (4.2.8) lib/action_controller/metal.rb:237:in `block in action'
actionpack (4.2.8) lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
actionpack (4.2.8) lib/action_dispatch/routing/route_set.rb:43:in `serve'
actionpack (4.2.8) lib/action_dispatch/routing/mapper.rb:49:in `serve'
actionpack (4.2.8) lib/action_dispatch/journey/router.rb:43:in `block in serve'
actionpack (4.2.8) lib/action_dispatch/journey/router.rb:30:in `serve'
actionpack (4.2.8) lib/action_dispatch/routing/route_set.rb:817:in `call'
peek-performance_bar (1.3.0) lib/peek/views/performance_bar/process_utilization.rb:120:in `call'
lib/gitlab/middleware/multipart.rb:98:in `block in call'
lib/gitlab/middleware/multipart.rb:53:in `with_open_files'
lib/gitlab/middleware/multipart.rb:97:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:18:in `call'
flipper (0.10.2) lib/flipper/middleware/memoizer.rb:55:in `call'
lib/gitlab/etag_caching/middleware.rb:11:in `call'
warden (1.2.6) lib/warden/manager.rb:35:in `block in call'
warden (1.2.6) lib/warden/manager.rb:34:in `call'
rack-cors (0.4.0) lib/rack/cors.rb:80:in `call'
rack-attack (4.4.1) lib/rack/attack.rb:107:in `call'
rack (1.6.8) lib/rack/etag.rb:24:in `call'
rack (1.6.8) lib/rack/conditionalget.rb:38:in `call'
rack (1.6.8) lib/rack/head.rb:13:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/params_parser.rb:27:in `call'
lib/gitlab/middleware/read_only.rb:31:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/flash.rb:260:in `call'
rack (1.6.8) lib/rack/session/abstract/id.rb:225:in `context'
rack (1.6.8) lib/rack/session/abstract/id.rb:220:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/cookies.rb:560:in `call'
activerecord (4.2.8) lib/active_record/query_cache.rb:36:in `call'
activerecord (4.2.8) lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
activerecord (4.2.8) lib/active_record/migration.rb:377:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.2.8) lib/active_support/callbacks.rb:88:in `__run_callbacks__'
activesupport (4.2.8) lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
activesupport (4.2.8) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (4.2.8) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/reloader.rb:73:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
better_errors (2.1.1) lib/better_errors/middleware.rb:84:in `protected_app_call'
better_errors (2.1.1) lib/better_errors/middleware.rb:79:in `better_errors_call'
better_errors (2.1.1) lib/better_errors/middleware.rb:57:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
railties (4.2.8) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.2.8) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.2.8) lib/active_support/tagged_logging.rb:68:in `block in tagged'
activesupport (4.2.8) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (4.2.8) lib/active_support/tagged_logging.rb:68:in `tagged'
railties (4.2.8) lib/rails/rack/logger.rb:20:in `call'
sprockets-rails (3.2.0) lib/sprockets/rails/quiet_assets.rb:13:in `call'
lib/gitlab/request_context.rb:18:in `call'
request_store (1.3.1) lib/request_store/middleware.rb:9:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/request_id.rb:21:in `call'
rack (1.6.8) lib/rack/methodoverride.rb:22:in `call'
rack (1.6.8) lib/rack/runtime.rb:18:in `call'
rack (1.6.8) lib/rack/lock.rb:17:in `call'
actionpack (4.2.8) lib/action_dispatch/middleware/static.rb:120:in `call'
lib/gitlab/middleware/static.rb:9:in `call'
lib/gitlab/middleware/webpack_proxy.rb:20:in `perform_request'
rack-proxy (0.6.0) lib/rack/proxy.rb:57:in `call'
rack (1.6.8) lib/rack/sendfile.rb:113:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
sentry-raven (2.5.3) lib/raven/integrations/rack.rb:50:in `call'
railties (4.2.8) lib/rails/engine.rb:518:in `call'
railties (4.2.8) lib/rails/application.rb:165:in `call'
railties (4.2.8) lib/rails/railtie.rb:194:in `method_missing'
rack (1.6.8) lib/rack/urlmap.rb:66:in `block in call'
rack (1.6.8) lib/rack/urlmap.rb:50:in `call'
unicorn (5.1.0) lib/unicorn/http_server.rb:562:in `process_client'
unicorn (5.1.0) lib/unicorn/http_server.rb:658:in `worker_loop'
unicorn (5.1.0) lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
unicorn (5.1.0) lib/unicorn/http_server.rb:132:in `start'
unicorn (5.1.0) bin/unicorn_rails:209:in `<top (required)>'
/Users/monozok/.rbenv/versions/2.3.5/bin/unicorn_rails:22:in `<top (required)>'
bundler (1.16.0) lib/bundler/cli/exec.rb:75:in `kernel_load'
bundler (1.16.0) lib/bundler/cli/exec.rb:28:in `run'
bundler (1.16.0) lib/bundler/cli.rb:424:in `exec'
bundler (1.16.0) lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
bundler (1.16.0) lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
bundler (1.16.0) lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
bundler (1.16.0) lib/bundler/cli.rb:27:in `dispatch'
bundler (1.16.0) lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
bundler (1.16.0) lib/bundler/cli.rb:18:in `start'
bundler (1.16.0) exe/bundle:30:in `block in <top (required)>'
bundler (1.16.0) lib/bundler/friendly_errors.rb:122:in `with_friendly_errors'
bundler (1.16.0) exe/bundle:22:in `<top (required)>'
/Users/monozok/.rbenv/versions/2.3.5/bin/bundle:22:in `<main>'
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)